None
This notebook allows visual inspection of a set of dithered images that are combined as part of calwebb_image3. The notebook will take a set of four simulated images, starting with the uncal format files that are the output of MIRISim. These images will have a randomized location for about 50 point sources and no extended sources.
These data files will be processed through calwebb_detector1, calwebb_image2 and calwebb_image3 and the output *i2d data file will be inspected. Past versions of the pipeline had the background subtracted twice, so the background levels were strongly negative, and other versions have had the background pixels wrongly flagged by outlier_detection. Part of this notebook is simply a visual inspection to be sure there is nothing obviously wrong with the output. While this notebook will look at the output of calwebb_image3 as a whole, the main task is to find where outlier_detection is incorrectly flagging pixels.
This notebook also inserts 'cr hits' at various random locations in each cal image (output of calwebb_image2), before running them through calwebb_image3. This notebook then tests whether the cr hits are correctly flagged as outliers by the outlier_detection step. There is also a visual comparison between the combined data with and without outlier_detection being performed to see if the output image has fewer artifacts in the version with outlier_detection.
Pipeline documentation: https://jwst-pipeline.readthedocs.io/en/latest/jwst/outlier_detection/main.html
Pipeline code: https://github.com/spacetelescope/jwst/tree/master/jwst/outlier_detection
The outlier detection algorithm is defined in the confluence page listed here. It is used to reject outlier pixels found when comparing multiple dithered images together. Images are compared in overlapping regions and any pixel that seems to be an outlier (based on iterative sigma clipping) can be flagged and rejected from any combined set or mosaic of images.
https://outerspace.stsci.edu/display/JWSTCC/Vanilla+Outlier+Detection
Here are some of the terms that will be used in this notebook.
JWST: James Webb Space Telescope
MIRI: Mid-Infrafred Instrument
MIRISim: MIRI data simulator
This test takes a set of simulated images (at different dithered positions) and proceses them through all three stages of the Imager pipeline: calwebb_detector1, calwebb_image2, and calwebb_image3. A set of random pixels have added simulated 'cr hits' and this notebook checks that those pixels are properly flagged as outliers. The tests being run here also look at the output of the full image3 pipeine (the combined i2d data), check that the source_catalog output catalog marks the locations of our point sources, and checks on the number of pixels that are flagged in the outlier_detection step to be sure that the step is not flagging too many pixels. There is also a comparison between the images run through calwebb_image3 with and without outlier_detection performed to see if the outliers were detected and removed from the image as they should be with outlier_detection.
The set of data used in this particular test were created with the MIRI Data Simulator (MIRISim). The simulator created four imaging mode files, one exposure each at four different dither positions, using the specified filter. There are approximately 50 point sources scattered through the images.
The following cell sets up a temporary directory (using python's tempfile.TemporaryDirectory()), and changes the script's active directory into that directory (using python's os.chdir()). This is so that, when the notebook is run through, it will download files to (and create output files in) the temporary directory rather than in the notebook's directory. This makes cleanup significantly easier (since all output files are deleted when the notebook is shut down), and also means that different notebooks in the same directory won't interfere with each other when run by the automated webpage generation process.
If you want the notebook to generate output in the notebook's directory, simply don't run this cell.
If you have a file (or files) that are kept in the notebook's directory, and that the notebook needs to use while running, you can copy that file into the directory (the code to do so is present below, but commented out).
# Create a temporary directory to hold notebook output, and change the working directory to that directory.
from tempfile import TemporaryDirectory
import os
import shutil
data_dir = TemporaryDirectory()
# If you have files that are in the notebook's directory, but that the notebook will need to use while
# running, copy them into the temporary directory here.
#
# files = ['name_of_file']
# for file_name in files:
# shutil.copy(file_name, os.path.join(data_dir.name, file_name))
os.chdir(data_dir.name)
print(data_dir.name)
/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao
import os
if 'CRDS_CACHE_TYPE' in os.environ:
if os.environ['CRDS_CACHE_TYPE'] == 'local':
os.environ['CRDS_PATH'] = os.path.join(os.environ['HOME'], 'crds', 'cache')
elif os.path.isdir(os.environ['CRDS_CACHE_TYPE']):
os.environ['CRDS_PATH'] = os.environ['CRDS_CACHE_TYPE']
print('CRDS cache location: {}'.format(os.environ['CRDS_PATH']))
CRDS cache location: /grp/crds/cache
List the package imports and why they are relevant to this notebook.
from astropy.io import fits
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval, LinearStretch
from astropy import table
from astropy.visualization import (MinMaxInterval, SqrtStretch, ImageNormalize)
from ci_watson.artifactory_helpers import get_bigdata
import glob
from jwst.pipeline import Detector1Pipeline, Image2Pipeline, Image3Pipeline
from jwst import associations
from jwst.associations.lib.rules_level3_base import DMS_Level3_Base
from jwst.associations import asn_from_list
from jwst.datamodels import RampModel, ImageModel, dqflags
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
import numpy as np
import os
import random
input_files = ['starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits',
'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits',
'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits',
'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits']
for file in input_files:
input_file = get_bigdata('jwst_validation_notebooks',
'validation_data',
'outlier_detection',
'outlier_detection_miri_test',
file)
#This readnoise file is needed for use with simulated data which has higher readnoise than actual data.
readnoise = get_bigdata('jwst_validation_notebooks',
'validation_data',
'jump',
'jump_miri_test',
'jwst_mirisim_readnoise.fits')
print("Finished Downloads")
Finished Downloads
The sections below will run the data through multiple pipeline steps, starting with calwebb_detector1 and calwebb_image2.
# Run the calwebb_detector1 pipeline
# set up pipeline parameters
rej_thresh=10.0 # rejection threshold for jump step
print('There are ', len(input_files), ' images.')
slopelist = []
# loop over list of files
for file in input_files:
# set up pipeline parameters for input
pipe1 = Detector1Pipeline()
pipe1.jump.rejection_threshold = rej_thresh
pipe1.jump.override_readnoise = readnoise
pipe1.ramp_fit.override_readnoise = readnoise
pipe1.refpix.skip = True # needs update to simulator for this to work properly with simulated data
# set up output file name
base, remainder = file.split('.')
outname = base
pipe1.output_file = outname+'.fits'
# Run pipeline on each file
rampfile = pipe1.run(file)
slopelist.append(rampfile)
# Close the input files
#file.close()
print('Detector 1 steps completed on all files.')
print(slopelist)
2022-10-06 01:00:37,729 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2022-10-06 01:00:37,730 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2022-10-06 01:00:37,731 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2022-10-06 01:00:37,732 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2022-10-06 01:00:37,733 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2022-10-06 01:00:37,734 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2022-10-06 01:00:37,735 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2022-10-06 01:00:37,736 - stpipe.Detector1Pipeline.rscd - INFO - RscdStep instance created.
2022-10-06 01:00:37,737 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2022-10-06 01:00:37,738 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2022-10-06 01:00:37,740 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2022-10-06 01:00:37,741 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2022-10-06 01:00:37,742 - stpipe.Detector1Pipeline.reset - INFO - ResetStep instance created.
2022-10-06 01:00:37,743 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2022-10-06 01:00:37,744 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2022-10-06 01:00:37,746 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2022-10-06 01:00:37,747 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2022-10-06 01:00:37,916 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits',).
There are 4 images.
2022-10-06 01:00:37,926 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'reset': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2022-10-06 01:00:39,288 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits' reftypes = ['dark', 'gain', 'ipc', 'linearity', 'mask', 'persat', 'reset', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2022-10-06 01:00:39,728 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits'.
2022-10-06 01:00:39,730 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits'.
2022-10-06 01:00:39,732 - stpipe.Detector1Pipeline - INFO - Prefetch for IPC reference file is 'N/A'.
2022-10-06 01:00:39,732 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits'.
2022-10-06 01:00:39,733 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits'.
2022-10-06 01:00:39,735 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2022-10-06 01:00:39,735 - stpipe.Detector1Pipeline - INFO - Override for READNOISE reference file is '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits'.
2022-10-06 01:00:39,736 - stpipe.Detector1Pipeline - INFO - Prefetch for RESET reference file is '/grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits'.
2022-10-06 01:00:39,737 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is '/grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits'.
2022-10-06 01:00:39,738 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits'.
2022-10-06 01:00:39,739 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is 'N/A'.
2022-10-06 01:00:39,740 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2022-10-06 01:00:39,740 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2022-10-06 01:00:39,741 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2022-10-06 01:00:40,279 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:40,281 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:40,723 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES=1 is a power of 2; correction not needed
2022-10-06 01:00:40,724 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2022-10-06 01:00:40,728 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2022-10-06 01:00:40,871 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:40,872 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:40,895 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits
2022-10-06 01:00:41,474 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2022-10-06 01:00:41,607 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:41,608 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}
2022-10-06 01:00:41,631 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits
2022-10-06 01:00:45,566 - stpipe.Detector1Pipeline.saturation - INFO - Detected 1071 saturated pixels
2022-10-06 01:00:45,631 - stpipe.Detector1Pipeline.saturation - INFO - Detected 0 A/D floor pixels
2022-10-06 01:00:45,643 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2022-10-06 01:00:45,795 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:45,797 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:45,820 - stpipe.Detector1Pipeline.ipc - INFO - Using IPC reference file N/A
2022-10-06 01:00:45,821 - stpipe.Detector1Pipeline.ipc - WARNING - No IPC reference file found
2022-10-06 01:00:45,821 - stpipe.Detector1Pipeline.ipc - WARNING - IPC step will be skipped
2022-10-06 01:00:46,167 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2022-10-06 01:00:46,278 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:46,280 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:46,628 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe done
2022-10-06 01:00:46,740 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:46,742 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:47,095 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe done
2022-10-06 01:00:47,229 - stpipe.Detector1Pipeline.reset - INFO - Step reset running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:47,231 - stpipe.Detector1Pipeline.reset - INFO - Step reset parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:47,255 - stpipe.Detector1Pipeline.reset - INFO - Using RESET reference file /grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits
2022-10-06 01:00:49,387 - stpipe.Detector1Pipeline.reset - INFO - Step reset done
2022-10-06 01:00:49,510 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:49,512 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:00:49,536 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits
2022-10-06 01:00:51,625 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2022-10-06 01:00:51,799 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:51,801 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}
2022-10-06 01:00:51,824 - stpipe.Detector1Pipeline.rscd - INFO - Using RSCD reference file /grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits
2022-10-06 01:00:52,218 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd done
2022-10-06 01:00:52,409 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:00:52,410 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}
2022-10-06 01:00:52,435 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits
2022-10-06 01:01:12,714 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=3, ngroups=25, nframes=1, groupgap=0
2022-10-06 01:01:12,715 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=2, ngroups=200, nframes=1, groupgap=0
2022-10-06 01:01:14,254 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2022-10-06 01:01:14,420 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:01:14,421 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2022-10-06 01:01:14,422 - stpipe.Detector1Pipeline.refpix - INFO - Step skipped.
2022-10-06 01:01:14,425 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2022-10-06 01:01:14,531 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:01:14,533 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}
2022-10-06 01:01:14,543 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 10 sigma
2022-10-06 01:01:14,559 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:01:14,612 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:01:15,192 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2022-10-06 01:01:15,345 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 1:
2022-10-06 01:01:17,957 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2982 pixels with at least one CR from five or more groups.
2022-10-06 01:01:17,958 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:01:17,958 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:01:18,586 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 2:
2022-10-06 01:01:21,248 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2445 pixels with at least one CR from five or more groups.
2022-10-06 01:01:21,249 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:01:21,250 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:01:21,799 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 3:
2022-10-06 01:01:24,459 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2541 pixels with at least one CR from five or more groups.
2022-10-06 01:01:24,460 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:01:24,461 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:01:25,020 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 9.82735 sec
2022-10-06 01:01:25,023 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 10.480309
2022-10-06 01:01:25,030 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2022-10-06 01:01:25,154 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:01:25,156 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}
2022-10-06 01:01:25,179 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:01:25,180 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:01:25,233 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2022-10-06 01:01:25,234 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
2022-10-06 01:01:25,574 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of leading groups that are flagged as DO_NOT_USE: 1
2022-10-06 01:01:25,576 - stpipe.Detector1Pipeline.ramp_fit - INFO - MIRI dataset has all pixels in the final group flagged as DO_NOT_USE.
2022-10-06 01:02:01,072 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 23
2022-10-06 01:02:01,074 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 3
2022-10-06 01:02:01,180 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2022-10-06 01:02:01,316 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:01,318 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:01,365 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:02:01,366 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:02:01,371 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:02:01,480 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(3, 1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:01,482 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:01,524 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:02:01,525 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:02:01,530 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:02:01,651 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rateints.fits
2022-10-06 01:02:01,652 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2022-10-06 01:02:01,652 - stpipe.Detector1Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:02:01,722 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits
2022-10-06 01:02:01,723 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
2022-10-06 01:02:01,731 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2022-10-06 01:02:01,732 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2022-10-06 01:02:01,733 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2022-10-06 01:02:01,735 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2022-10-06 01:02:01,736 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2022-10-06 01:02:01,737 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2022-10-06 01:02:01,738 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2022-10-06 01:02:01,739 - stpipe.Detector1Pipeline.rscd - INFO - RscdStep instance created.
2022-10-06 01:02:01,740 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2022-10-06 01:02:01,742 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2022-10-06 01:02:01,743 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2022-10-06 01:02:01,744 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2022-10-06 01:02:01,745 - stpipe.Detector1Pipeline.reset - INFO - ResetStep instance created.
2022-10-06 01:02:01,746 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2022-10-06 01:02:01,747 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2022-10-06 01:02:01,749 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2022-10-06 01:02:01,750 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2022-10-06 01:02:01,860 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits',).
2022-10-06 01:02:01,870 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'reset': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2022-10-06 01:02:02,247 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits' reftypes = ['dark', 'gain', 'ipc', 'linearity', 'mask', 'persat', 'reset', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2022-10-06 01:02:02,254 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits'.
2022-10-06 01:02:02,256 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits'.
2022-10-06 01:02:02,257 - stpipe.Detector1Pipeline - INFO - Prefetch for IPC reference file is 'N/A'.
2022-10-06 01:02:02,258 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits'.
2022-10-06 01:02:02,260 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits'.
2022-10-06 01:02:02,261 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2022-10-06 01:02:02,262 - stpipe.Detector1Pipeline - INFO - Override for READNOISE reference file is '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits'.
2022-10-06 01:02:02,262 - stpipe.Detector1Pipeline - INFO - Prefetch for RESET reference file is '/grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits'.
2022-10-06 01:02:02,264 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is '/grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits'.
2022-10-06 01:02:02,266 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits'.
2022-10-06 01:02:02,267 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is 'N/A'.
2022-10-06 01:02:02,267 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2022-10-06 01:02:02,268 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2022-10-06 01:02:02,268 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2022-10-06 01:02:02,779 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:02,781 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:03,119 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES=1 is a power of 2; correction not needed
2022-10-06 01:02:03,120 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2022-10-06 01:02:03,123 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2022-10-06 01:02:03,237 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:03,238 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:03,261 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits
2022-10-06 01:02:03,770 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2022-10-06 01:02:03,883 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:03,885 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}
2022-10-06 01:02:03,908 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits
2022-10-06 01:02:07,698 - stpipe.Detector1Pipeline.saturation - INFO - Detected 1100 saturated pixels
2022-10-06 01:02:07,757 - stpipe.Detector1Pipeline.saturation - INFO - Detected 0 A/D floor pixels
2022-10-06 01:02:07,766 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2022-10-06 01:02:07,879 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:07,880 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:07,901 - stpipe.Detector1Pipeline.ipc - INFO - Using IPC reference file N/A
2022-10-06 01:02:07,902 - stpipe.Detector1Pipeline.ipc - WARNING - No IPC reference file found
2022-10-06 01:02:07,903 - stpipe.Detector1Pipeline.ipc - WARNING - IPC step will be skipped
2022-10-06 01:02:08,246 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2022-10-06 01:02:08,354 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:08,356 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:08,686 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe done
2022-10-06 01:02:08,792 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:08,793 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:09,141 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe done
2022-10-06 01:02:09,249 - stpipe.Detector1Pipeline.reset - INFO - Step reset running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:09,251 - stpipe.Detector1Pipeline.reset - INFO - Step reset parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:09,274 - stpipe.Detector1Pipeline.reset - INFO - Using RESET reference file /grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits
2022-10-06 01:02:09,895 - stpipe.Detector1Pipeline.reset - INFO - Step reset done
2022-10-06 01:02:10,007 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:10,008 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:02:10,030 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits
2022-10-06 01:02:11,377 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2022-10-06 01:02:11,486 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:11,487 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}
2022-10-06 01:02:11,510 - stpipe.Detector1Pipeline.rscd - INFO - Using RSCD reference file /grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits
2022-10-06 01:02:11,905 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd done
2022-10-06 01:02:12,049 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:12,051 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}
2022-10-06 01:02:12,073 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits
2022-10-06 01:02:13,571 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=3, ngroups=25, nframes=1, groupgap=0
2022-10-06 01:02:13,572 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=2, ngroups=200, nframes=1, groupgap=0
2022-10-06 01:02:14,886 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2022-10-06 01:02:15,008 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:15,010 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2022-10-06 01:02:15,010 - stpipe.Detector1Pipeline.refpix - INFO - Step skipped.
2022-10-06 01:02:15,013 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2022-10-06 01:02:15,124 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:15,125 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}
2022-10-06 01:02:15,136 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 10 sigma
2022-10-06 01:02:15,150 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:02:15,177 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:02:15,759 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2022-10-06 01:02:15,915 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 1:
2022-10-06 01:02:18,553 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2858 pixels with at least one CR from five or more groups.
2022-10-06 01:02:18,554 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:02:18,555 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:02:19,169 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 2:
2022-10-06 01:02:21,876 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2639 pixels with at least one CR from five or more groups.
2022-10-06 01:02:21,877 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:02:21,877 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:02:22,457 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 3:
2022-10-06 01:02:25,154 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2384 pixels with at least one CR from five or more groups.
2022-10-06 01:02:25,156 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:02:25,156 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:02:25,705 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 9.94521 sec
2022-10-06 01:02:25,709 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 10.572814
2022-10-06 01:02:25,715 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2022-10-06 01:02:25,854 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:02:25,856 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}
2022-10-06 01:02:25,881 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:02:25,882 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:02:25,937 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2022-10-06 01:02:25,938 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
2022-10-06 01:02:26,274 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of leading groups that are flagged as DO_NOT_USE: 1
2022-10-06 01:02:26,276 - stpipe.Detector1Pipeline.ramp_fit - INFO - MIRI dataset has all pixels in the final group flagged as DO_NOT_USE.
2022-10-06 01:03:02,594 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 23
2022-10-06 01:03:02,596 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 3
2022-10-06 01:03:02,685 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2022-10-06 01:03:02,806 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:02,808 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:02,852 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:03:02,853 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:03:02,858 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:03:02,966 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(3, 1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:02,968 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:03,011 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:03:03,012 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:03:03,017 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:03:03,130 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rateints.fits
2022-10-06 01:03:03,131 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2022-10-06 01:03:03,132 - stpipe.Detector1Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:03:03,197 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits
2022-10-06 01:03:03,198 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
2022-10-06 01:03:03,206 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2022-10-06 01:03:03,207 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2022-10-06 01:03:03,208 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2022-10-06 01:03:03,210 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2022-10-06 01:03:03,211 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2022-10-06 01:03:03,212 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2022-10-06 01:03:03,213 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2022-10-06 01:03:03,214 - stpipe.Detector1Pipeline.rscd - INFO - RscdStep instance created.
2022-10-06 01:03:03,215 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2022-10-06 01:03:03,216 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2022-10-06 01:03:03,217 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2022-10-06 01:03:03,218 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2022-10-06 01:03:03,219 - stpipe.Detector1Pipeline.reset - INFO - ResetStep instance created.
2022-10-06 01:03:03,221 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2022-10-06 01:03:03,222 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2022-10-06 01:03:03,224 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2022-10-06 01:03:03,225 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2022-10-06 01:03:03,336 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits',).
2022-10-06 01:03:03,345 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'reset': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2022-10-06 01:03:03,719 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits' reftypes = ['dark', 'gain', 'ipc', 'linearity', 'mask', 'persat', 'reset', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2022-10-06 01:03:03,725 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits'.
2022-10-06 01:03:03,727 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits'.
2022-10-06 01:03:03,728 - stpipe.Detector1Pipeline - INFO - Prefetch for IPC reference file is 'N/A'.
2022-10-06 01:03:03,728 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits'.
2022-10-06 01:03:03,729 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits'.
2022-10-06 01:03:03,730 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2022-10-06 01:03:03,731 - stpipe.Detector1Pipeline - INFO - Override for READNOISE reference file is '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits'.
2022-10-06 01:03:03,731 - stpipe.Detector1Pipeline - INFO - Prefetch for RESET reference file is '/grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits'.
2022-10-06 01:03:03,733 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is '/grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits'.
2022-10-06 01:03:03,734 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits'.
2022-10-06 01:03:03,735 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is 'N/A'.
2022-10-06 01:03:03,736 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2022-10-06 01:03:03,736 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2022-10-06 01:03:03,736 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2022-10-06 01:03:04,209 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:04,210 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:04,553 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES=1 is a power of 2; correction not needed
2022-10-06 01:03:04,554 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2022-10-06 01:03:04,556 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2022-10-06 01:03:04,664 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:04,666 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:04,687 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits
2022-10-06 01:03:05,172 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2022-10-06 01:03:05,284 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:05,285 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}
2022-10-06 01:03:05,307 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits
2022-10-06 01:03:08,964 - stpipe.Detector1Pipeline.saturation - INFO - Detected 957 saturated pixels
2022-10-06 01:03:09,026 - stpipe.Detector1Pipeline.saturation - INFO - Detected 0 A/D floor pixels
2022-10-06 01:03:09,035 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2022-10-06 01:03:09,148 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:09,150 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:09,171 - stpipe.Detector1Pipeline.ipc - INFO - Using IPC reference file N/A
2022-10-06 01:03:09,172 - stpipe.Detector1Pipeline.ipc - WARNING - No IPC reference file found
2022-10-06 01:03:09,172 - stpipe.Detector1Pipeline.ipc - WARNING - IPC step will be skipped
2022-10-06 01:03:09,516 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2022-10-06 01:03:09,626 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:09,628 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:09,976 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe done
2022-10-06 01:03:10,088 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:10,090 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:10,440 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe done
2022-10-06 01:03:10,562 - stpipe.Detector1Pipeline.reset - INFO - Step reset running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:10,563 - stpipe.Detector1Pipeline.reset - INFO - Step reset parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:10,587 - stpipe.Detector1Pipeline.reset - INFO - Using RESET reference file /grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits
2022-10-06 01:03:11,221 - stpipe.Detector1Pipeline.reset - INFO - Step reset done
2022-10-06 01:03:11,337 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:11,339 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:03:11,362 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits
2022-10-06 01:03:12,708 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2022-10-06 01:03:12,842 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:12,844 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}
2022-10-06 01:03:12,868 - stpipe.Detector1Pipeline.rscd - INFO - Using RSCD reference file /grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits
2022-10-06 01:03:13,257 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd done
2022-10-06 01:03:13,374 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:13,376 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}
2022-10-06 01:03:13,399 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits
2022-10-06 01:03:14,907 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=3, ngroups=25, nframes=1, groupgap=0
2022-10-06 01:03:14,908 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=2, ngroups=200, nframes=1, groupgap=0
2022-10-06 01:03:16,208 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2022-10-06 01:03:16,333 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:16,335 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2022-10-06 01:03:16,335 - stpipe.Detector1Pipeline.refpix - INFO - Step skipped.
2022-10-06 01:03:16,338 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2022-10-06 01:03:16,446 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:16,448 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}
2022-10-06 01:03:16,458 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 10 sigma
2022-10-06 01:03:16,472 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:03:16,497 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:03:17,080 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2022-10-06 01:03:17,235 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 1:
2022-10-06 01:03:19,833 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2601 pixels with at least one CR from five or more groups.
2022-10-06 01:03:19,836 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:03:19,836 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:03:20,399 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 2:
2022-10-06 01:03:23,073 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2274 pixels with at least one CR from five or more groups.
2022-10-06 01:03:23,074 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:03:23,075 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:03:23,595 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 3:
2022-10-06 01:03:26,284 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2368 pixels with at least one CR from five or more groups.
2022-10-06 01:03:26,286 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:03:26,286 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:03:26,816 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 9.73479 sec
2022-10-06 01:03:26,820 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 10.361533
2022-10-06 01:03:26,825 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2022-10-06 01:03:26,951 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:03:26,953 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}
2022-10-06 01:03:26,978 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:03:26,979 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:03:27,033 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2022-10-06 01:03:27,033 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
2022-10-06 01:03:27,373 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of leading groups that are flagged as DO_NOT_USE: 1
2022-10-06 01:03:27,375 - stpipe.Detector1Pipeline.ramp_fit - INFO - MIRI dataset has all pixels in the final group flagged as DO_NOT_USE.
2022-10-06 01:04:01,805 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 23
2022-10-06 01:04:01,806 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 3
2022-10-06 01:04:01,900 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2022-10-06 01:04:02,037 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:02,039 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:02,079 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:04:02,080 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:04:02,085 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:04:02,189 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(3, 1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:02,191 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:02,234 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:04:02,234 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:04:02,239 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:04:02,352 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rateints.fits
2022-10-06 01:04:02,353 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2022-10-06 01:04:02,354 - stpipe.Detector1Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:04:02,420 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits
2022-10-06 01:04:02,421 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
2022-10-06 01:04:02,429 - stpipe.Detector1Pipeline - INFO - Detector1Pipeline instance created.
2022-10-06 01:04:02,430 - stpipe.Detector1Pipeline.group_scale - INFO - GroupScaleStep instance created.
2022-10-06 01:04:02,431 - stpipe.Detector1Pipeline.dq_init - INFO - DQInitStep instance created.
2022-10-06 01:04:02,432 - stpipe.Detector1Pipeline.saturation - INFO - SaturationStep instance created.
2022-10-06 01:04:02,433 - stpipe.Detector1Pipeline.ipc - INFO - IPCStep instance created.
2022-10-06 01:04:02,434 - stpipe.Detector1Pipeline.superbias - INFO - SuperBiasStep instance created.
2022-10-06 01:04:02,435 - stpipe.Detector1Pipeline.refpix - INFO - RefPixStep instance created.
2022-10-06 01:04:02,436 - stpipe.Detector1Pipeline.rscd - INFO - RscdStep instance created.
2022-10-06 01:04:02,437 - stpipe.Detector1Pipeline.firstframe - INFO - FirstFrameStep instance created.
2022-10-06 01:04:02,439 - stpipe.Detector1Pipeline.lastframe - INFO - LastFrameStep instance created.
2022-10-06 01:04:02,440 - stpipe.Detector1Pipeline.linearity - INFO - LinearityStep instance created.
2022-10-06 01:04:02,441 - stpipe.Detector1Pipeline.dark_current - INFO - DarkCurrentStep instance created.
2022-10-06 01:04:02,442 - stpipe.Detector1Pipeline.reset - INFO - ResetStep instance created.
2022-10-06 01:04:02,443 - stpipe.Detector1Pipeline.persistence - INFO - PersistenceStep instance created.
2022-10-06 01:04:02,444 - stpipe.Detector1Pipeline.jump - INFO - JumpStep instance created.
2022-10-06 01:04:02,446 - stpipe.Detector1Pipeline.ramp_fit - INFO - RampFitStep instance created.
2022-10-06 01:04:02,447 - stpipe.Detector1Pipeline.gain_scale - INFO - GainScaleStep instance created.
2022-10-06 01:04:02,554 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline running with args ('starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits',).
2022-10-06 01:04:02,564 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_calibrated_ramp': False, 'steps': {'group_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dq_init': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'saturation': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}, 'ipc': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'superbias': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'refpix': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}, 'rscd': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}, 'firstframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'lastframe': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'linearity': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'dark_current': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}, 'reset': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}, 'persistence': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'input_trapsfilled': '', 'flag_pers_cutoff': 40.0, 'save_persistence': False, 'save_trapsfilled': True}, 'jump': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}, 'ramp_fit': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}, 'gain_scale': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}}}
2022-10-06 01:04:02,935 - stpipe.Detector1Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits' reftypes = ['dark', 'gain', 'ipc', 'linearity', 'mask', 'persat', 'reset', 'rscd', 'saturation', 'superbias', 'trapdensity', 'trappars']
2022-10-06 01:04:02,945 - stpipe.Detector1Pipeline - INFO - Prefetch for DARK reference file is '/grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits'.
2022-10-06 01:04:02,947 - stpipe.Detector1Pipeline - INFO - Prefetch for GAIN reference file is '/grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits'.
2022-10-06 01:04:02,948 - stpipe.Detector1Pipeline - INFO - Prefetch for IPC reference file is 'N/A'.
2022-10-06 01:04:02,948 - stpipe.Detector1Pipeline - INFO - Prefetch for LINEARITY reference file is '/grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits'.
2022-10-06 01:04:02,950 - stpipe.Detector1Pipeline - INFO - Prefetch for MASK reference file is '/grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits'.
2022-10-06 01:04:02,951 - stpipe.Detector1Pipeline - INFO - Prefetch for PERSAT reference file is 'N/A'.
2022-10-06 01:04:02,952 - stpipe.Detector1Pipeline - INFO - Override for READNOISE reference file is '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits'.
2022-10-06 01:04:02,952 - stpipe.Detector1Pipeline - INFO - Prefetch for RESET reference file is '/grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits'.
2022-10-06 01:04:02,954 - stpipe.Detector1Pipeline - INFO - Prefetch for RSCD reference file is '/grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits'.
2022-10-06 01:04:02,955 - stpipe.Detector1Pipeline - INFO - Prefetch for SATURATION reference file is '/grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits'.
2022-10-06 01:04:02,956 - stpipe.Detector1Pipeline - INFO - Prefetch for SUPERBIAS reference file is 'N/A'.
2022-10-06 01:04:02,957 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPDENSITY reference file is 'N/A'.
2022-10-06 01:04:02,957 - stpipe.Detector1Pipeline - INFO - Prefetch for TRAPPARS reference file is 'N/A'.
2022-10-06 01:04:02,958 - stpipe.Detector1Pipeline - INFO - Starting calwebb_detector1 ...
2022-10-06 01:04:03,421 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:03,423 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:03,756 - stpipe.Detector1Pipeline.group_scale - INFO - NFRAMES=1 is a power of 2; correction not needed
2022-10-06 01:04:03,756 - stpipe.Detector1Pipeline.group_scale - INFO - Step will be skipped
2022-10-06 01:04:03,759 - stpipe.Detector1Pipeline.group_scale - INFO - Step group_scale done
2022-10-06 01:04:03,870 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:03,872 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:03,894 - stpipe.Detector1Pipeline.dq_init - INFO - Using MASK reference file /grp/crds/cache/references/jwst/jwst_miri_mask_0028.fits
2022-10-06 01:04:04,386 - stpipe.Detector1Pipeline.dq_init - INFO - Step dq_init done
2022-10-06 01:04:04,500 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:04,502 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'n_pix_grow_sat': 1}
2022-10-06 01:04:04,525 - stpipe.Detector1Pipeline.saturation - INFO - Using SATURATION reference file /grp/crds/cache/references/jwst/jwst_miri_saturation_0026.fits
2022-10-06 01:04:08,398 - stpipe.Detector1Pipeline.saturation - INFO - Detected 971 saturated pixels
2022-10-06 01:04:08,461 - stpipe.Detector1Pipeline.saturation - INFO - Detected 0 A/D floor pixels
2022-10-06 01:04:08,469 - stpipe.Detector1Pipeline.saturation - INFO - Step saturation done
2022-10-06 01:04:08,585 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:08,586 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:08,607 - stpipe.Detector1Pipeline.ipc - INFO - Using IPC reference file N/A
2022-10-06 01:04:08,608 - stpipe.Detector1Pipeline.ipc - WARNING - No IPC reference file found
2022-10-06 01:04:08,608 - stpipe.Detector1Pipeline.ipc - WARNING - IPC step will be skipped
2022-10-06 01:04:08,953 - stpipe.Detector1Pipeline.ipc - INFO - Step ipc done
2022-10-06 01:04:09,072 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:09,073 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:09,426 - stpipe.Detector1Pipeline.firstframe - INFO - Step firstframe done
2022-10-06 01:04:09,546 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:09,547 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:09,902 - stpipe.Detector1Pipeline.lastframe - INFO - Step lastframe done
2022-10-06 01:04:10,021 - stpipe.Detector1Pipeline.reset - INFO - Step reset running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:10,022 - stpipe.Detector1Pipeline.reset - INFO - Step reset parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:10,046 - stpipe.Detector1Pipeline.reset - INFO - Using RESET reference file /grp/crds/cache/references/jwst/jwst_miri_reset_0047.fits
2022-10-06 01:04:10,678 - stpipe.Detector1Pipeline.reset - INFO - Step reset done
2022-10-06 01:04:10,801 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:10,803 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:04:10,827 - stpipe.Detector1Pipeline.linearity - INFO - Using Linearity reference file /grp/crds/cache/references/jwst/jwst_miri_linearity_0024.fits
2022-10-06 01:04:12,379 - stpipe.Detector1Pipeline.linearity - INFO - Step linearity done
2022-10-06 01:04:12,498 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:12,500 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'type': 'baseline'}
2022-10-06 01:04:12,523 - stpipe.Detector1Pipeline.rscd - INFO - Using RSCD reference file /grp/crds/cache/references/jwst/jwst_miri_rscd_0014.fits
2022-10-06 01:04:12,917 - stpipe.Detector1Pipeline.rscd - INFO - Step rscd done
2022-10-06 01:04:13,038 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:13,039 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'dark_output': None}
2022-10-06 01:04:13,063 - stpipe.Detector1Pipeline.dark_current - INFO - Using DARK reference file /grp/crds/cache/references/jwst/jwst_miri_dark_0058.fits
2022-10-06 01:04:14,580 - stpipe.Detector1Pipeline.dark_current - INFO - Science data nints=3, ngroups=25, nframes=1, groupgap=0
2022-10-06 01:04:14,581 - stpipe.Detector1Pipeline.dark_current - INFO - Dark data nints=2, ngroups=200, nframes=1, groupgap=0
2022-10-06 01:04:15,891 - stpipe.Detector1Pipeline.dark_current - INFO - Step dark_current done
2022-10-06 01:04:16,024 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:16,025 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'odd_even_columns': True, 'use_side_ref_pixels': True, 'side_smoothing_length': 11, 'side_gain': 1.0, 'odd_even_rows': True}
2022-10-06 01:04:16,026 - stpipe.Detector1Pipeline.refpix - INFO - Step skipped.
2022-10-06 01:04:16,028 - stpipe.Detector1Pipeline.refpix - INFO - Step refpix done
2022-10-06 01:04:16,140 - stpipe.Detector1Pipeline.jump - INFO - Step jump running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:16,142 - stpipe.Detector1Pipeline.jump - INFO - Step jump parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'rejection_threshold': 10.0, 'three_group_rejection_threshold': 6.0, 'four_group_rejection_threshold': 5.0, 'maximum_cores': 'none', 'flag_4_neighbors': True, 'max_jump_to_flag_neighbors': 1000.0, 'min_jump_to_flag_neighbors': 10.0, 'after_jump_flag_dn1': 0.0, 'after_jump_flag_time1': 0.0, 'after_jump_flag_dn2': 0.0, 'after_jump_flag_time2': 0.0}
2022-10-06 01:04:16,153 - stpipe.Detector1Pipeline.jump - INFO - CR rejection threshold = 10 sigma
2022-10-06 01:04:16,168 - stpipe.Detector1Pipeline.jump - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:04:16,193 - stpipe.Detector1Pipeline.jump - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:04:16,779 - stpipe.Detector1Pipeline.jump - INFO - Executing two-point difference method
2022-10-06 01:04:16,936 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 1:
2022-10-06 01:04:19,583 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2614 pixels with at least one CR from five or more groups.
2022-10-06 01:04:19,584 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:04:19,584 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:04:20,158 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 2:
2022-10-06 01:04:22,744 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2258 pixels with at least one CR from five or more groups.
2022-10-06 01:04:22,745 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:04:22,746 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:04:23,270 - stpipe.Detector1Pipeline.jump - INFO - Working on integration 3:
2022-10-06 01:04:25,935 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 2515 pixels with at least one CR from five or more groups.
2022-10-06 01:04:25,936 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from four groups.
2022-10-06 01:04:25,937 - stpipe.Detector1Pipeline.jump - INFO - From highest outlier, two-point found 0 pixels with at least one CR from three groups.
2022-10-06 01:04:26,513 - stpipe.Detector1Pipeline.jump - INFO - Total elapsed time = 9.7324 sec
2022-10-06 01:04:26,517 - stpipe.Detector1Pipeline.jump - INFO - The execution time in seconds: 10.364712
2022-10-06 01:04:26,523 - stpipe.Detector1Pipeline.jump - INFO - Step jump done
2022-10-06 01:04:26,670 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit running with args (<RampModel(3, 25, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:04:26,672 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'int_name': '', 'save_opt': False, 'opt_name': '', 'suppress_one_group': True, 'maximum_cores': 'none'}
2022-10-06 01:04:26,697 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using READNOISE reference file: /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/jwst_mirisim_readnoise.fits
2022-10-06 01:04:26,697 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using GAIN reference file: /grp/crds/cache/references/jwst/jwst_miri_gain_0008.fits
2022-10-06 01:04:26,750 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using algorithm = ols
2022-10-06 01:04:26,750 - stpipe.Detector1Pipeline.ramp_fit - INFO - Using weighting = optimal
2022-10-06 01:04:27,091 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of leading groups that are flagged as DO_NOT_USE: 1
2022-10-06 01:04:27,093 - stpipe.Detector1Pipeline.ramp_fit - INFO - MIRI dataset has all pixels in the final group flagged as DO_NOT_USE.
2022-10-06 01:05:03,852 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of groups per integration: 23
2022-10-06 01:05:03,854 - stpipe.Detector1Pipeline.ramp_fit - INFO - Number of integrations: 3
2022-10-06 01:05:03,951 - stpipe.Detector1Pipeline.ramp_fit - INFO - Step ramp_fit done
2022-10-06 01:05:04,098 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:05:04,100 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scale', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:05:04,145 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:05:04,146 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:05:04,151 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:05:04,264 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale running with args (<CubeModel(3, 1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1.fits>,).
2022-10-06 01:05:04,265 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'gain_scaleints', 'search_output_file': True, 'input_dir': ''}
2022-10-06 01:05:04,309 - stpipe.Detector1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file
2022-10-06 01:05:04,309 - stpipe.Detector1Pipeline.gain_scale - INFO - Step will be skipped
2022-10-06 01:05:04,314 - stpipe.Detector1Pipeline.gain_scale - INFO - Step gain_scale done
2022-10-06 01:05:04,423 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rateints.fits
2022-10-06 01:05:04,424 - stpipe.Detector1Pipeline - INFO - ... ending calwebb_detector1
2022-10-06 01:05:04,424 - stpipe.Detector1Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:05:04,491 - stpipe.Detector1Pipeline - INFO - Saved model in starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits
2022-10-06 01:05:04,491 - stpipe.Detector1Pipeline - INFO - Step Detector1Pipeline done
Detector 1 steps completed on all files. [<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits>, <ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits>, <ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits>, <ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits>]
# Run Calwebb_image2 on output files from detector1
print('There are ', len(slopelist), ' images.')
# create an object for the pipeline
callist = []
ratefilenames = []
# cycle through files
for rampfile in slopelist:
pipe2 = Image2Pipeline()
filename = rampfile.meta.filename
ratefilenames.append(filename)
# Set pipeline parameters
pipe2.save_results = True
pipe2.output_file = filename +'_cal.fits'
pipe2.resample.save_results = True
pipe2.suffix = None
calfile = pipe2.run(rampfile)
callist.append(calfile)
print(callist)
2022-10-06 01:05:04,506 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-10-06 01:05:04,508 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-10-06 01:05:04,509 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-10-06 01:05:04,511 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-10-06 01:05:04,512 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-10-06 01:05:04,513 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-10-06 01:05:04,628 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:04,633 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}}}
2022-10-06 01:05:04,641 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-10-06 01:05:04,655 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-10-06 01:05:04,657 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-10-06 01:05:04,657 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-10-06 01:05:04,658 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-10-06 01:05:04,658 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-10-06 01:05:04,658 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-10-06 01:05:04,661 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-10-06 01:05:04,662 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-10-06 01:05:04,663 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-10-06 01:05:04,665 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0626.fits'.
2022-10-06 01:05:04,667 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-10-06 01:05:04,667 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-10-06 01:05:04,667 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-10-06 01:05:04,668 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-10-06 01:05:04,668 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-10-06 01:05:04,668 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-10-06 01:05:04,669 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-10-06 01:05:04,670 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-10-06 01:05:04,673 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-10-06 01:05:04,673 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-10-06 01:05:04,674 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-10-06 01:05:04,674 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-10-06 01:05:04,674 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-10-06 01:05:04,675 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
There are 4 images.
2022-10-06 01:05:04,710 - stpipe.Image2Pipeline - INFO - Processing product starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:04,711 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits> ...
2022-10-06 01:05:04,826 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:04,828 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-10-06 01:05:05,049 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-10-06 01:05:05,127 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.019532876 -0.016147752 0.021999558 0.015037465 359.990697171 0.017630270 359.987889494 -0.013568846
2022-10-06 01:05:05,128 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.019532876 -0.016147752 0.021999558 0.015037465 359.990697171 0.017630270 359.987889494 -0.013568846
2022-10-06 01:05:05,129 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-10-06 01:05:05,162 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-10-06 01:05:05,173 - stpipe.Image2Pipeline.assign_wcs - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:05,173 - stpipe.Image2Pipeline.assign_wcs - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:05,186 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-10-06 01:05:05,301 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:05,302 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-10-06 01:05:05,804 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:05,805 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:05,806 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:05,815 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:05,951 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-10-06 01:05:06,080 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:06,082 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-06 01:05:06,106 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-10-06 01:05:06,106 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-10-06 01:05:06,191 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-10-06 01:05:06,192 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-10-06 01:05:06,192 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-10-06 01:05:06,193 - stpipe.Image2Pipeline.photom - INFO - filter: F1130W
2022-10-06 01:05:06,292 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-10-06 01:05:06,294 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-10-06 01:05:06,295 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-10-06 01:05:06,296 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 2.54947
2022-10-06 01:05:06,310 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-10-06 01:05:06,427 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:06,429 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}
2022-10-06 01:05:06,456 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-10-06 01:05:06,479 - stpipe.Image2Pipeline.resample - INFO - Driz parameter kernel: square
2022-10-06 01:05:06,479 - stpipe.Image2Pipeline.resample - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:05:06,480 - stpipe.Image2Pipeline.resample - INFO - Driz parameter fillval: INDEF
2022-10-06 01:05:06,481 - stpipe.Image2Pipeline.resample - INFO - Driz parameter weight_type: ivm
2022-10-06 01:05:06,596 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-10-06 01:05:07,222 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:07,407 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-10-06 01:05:08,117 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:08,312 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-10-06 01:05:08,968 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:09,248 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-10-06 01:05:09,855 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:10,057 - stpipe.Image2Pipeline.resample - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/resample/resample.py:308: RuntimeWarning: divide by zero encountered in reciprocal
output_variance = np.reciprocal(inverse_variance_sum)
2022-10-06 01:05:10,090 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.019529633 -0.016360379 0.022280129 0.014982155 359.990661583 0.017756872 359.987911088 -0.013585662
2022-10-06 01:05:10,231 - stpipe.Image2Pipeline.resample - INFO - Saved model in starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_i2d.fits
2022-10-06 01:05:10,232 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-10-06 01:05:10,233 - stpipe.Image2Pipeline - INFO - Finished processing product starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:10,234 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-10-06 01:05:10,235 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:05:10,408 - stpipe.Image2Pipeline - INFO - Saved model in starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits
2022-10-06 01:05:10,409 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
2022-10-06 01:05:10,414 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-10-06 01:05:10,415 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-10-06 01:05:10,416 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-10-06 01:05:10,417 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-10-06 01:05:10,419 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-10-06 01:05:10,420 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-10-06 01:05:10,550 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:10,555 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}}}
2022-10-06 01:05:10,563 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-10-06 01:05:10,571 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-10-06 01:05:10,572 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-10-06 01:05:10,573 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-10-06 01:05:10,573 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-10-06 01:05:10,574 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-10-06 01:05:10,574 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-10-06 01:05:10,575 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-10-06 01:05:10,576 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-10-06 01:05:10,577 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-10-06 01:05:10,578 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0626.fits'.
2022-10-06 01:05:10,579 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-10-06 01:05:10,580 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-10-06 01:05:10,580 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-10-06 01:05:10,581 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-10-06 01:05:10,581 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-10-06 01:05:10,582 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-10-06 01:05:10,582 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-10-06 01:05:10,583 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-10-06 01:05:10,584 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-10-06 01:05:10,585 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-10-06 01:05:10,585 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-10-06 01:05:10,586 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-10-06 01:05:10,586 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-10-06 01:05:10,587 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-10-06 01:05:10,637 - stpipe.Image2Pipeline - INFO - Processing product starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:10,638 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits> ...
2022-10-06 01:05:10,778 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:10,779 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-10-06 01:05:10,990 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-10-06 01:05:11,070 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.020172025 -0.017469545 0.022638707 0.013715672 359.991336319 0.016308478 359.988528642 -0.014890638
2022-10-06 01:05:11,070 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.020172025 -0.017469545 0.022638707 0.013715672 359.991336319 0.016308478 359.988528642 -0.014890638
2022-10-06 01:05:11,071 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-10-06 01:05:11,106 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-10-06 01:05:11,118 - stpipe.Image2Pipeline.assign_wcs - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:11,119 - stpipe.Image2Pipeline.assign_wcs - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:11,134 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-10-06 01:05:11,264 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:11,265 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-10-06 01:05:11,382 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:11,383 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:11,384 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:11,391 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:11,527 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-10-06 01:05:11,656 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:11,657 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-06 01:05:11,683 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-10-06 01:05:11,684 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-10-06 01:05:11,772 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-10-06 01:05:11,774 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-10-06 01:05:11,774 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-10-06 01:05:11,774 - stpipe.Image2Pipeline.photom - INFO - filter: F1130W
2022-10-06 01:05:11,824 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-10-06 01:05:11,826 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-10-06 01:05:11,827 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-10-06 01:05:11,828 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 2.54947
2022-10-06 01:05:11,844 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-10-06 01:05:11,976 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:11,978 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}
2022-10-06 01:05:12,006 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-10-06 01:05:12,027 - stpipe.Image2Pipeline.resample - INFO - Driz parameter kernel: square
2022-10-06 01:05:12,028 - stpipe.Image2Pipeline.resample - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:05:12,029 - stpipe.Image2Pipeline.resample - INFO - Driz parameter fillval: INDEF
2022-10-06 01:05:12,029 - stpipe.Image2Pipeline.resample - INFO - Driz parameter weight_type: ivm
2022-10-06 01:05:12,155 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-10-06 01:05:12,862 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:13,065 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-10-06 01:05:13,744 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:13,980 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-10-06 01:05:14,593 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:14,789 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-10-06 01:05:15,395 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:15,592 - stpipe.Image2Pipeline.resample - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/resample/resample.py:308: RuntimeWarning: divide by zero encountered in reciprocal
output_variance = np.reciprocal(inverse_variance_sum)
2022-10-06 01:05:15,625 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.020168782 -0.017682172 0.022919277 0.013660362 359.991300731 0.016435079 359.988550236 -0.014907455
2022-10-06 01:05:15,797 - stpipe.Image2Pipeline.resample - INFO - Saved model in starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_i2d.fits
2022-10-06 01:05:15,798 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-10-06 01:05:15,798 - stpipe.Image2Pipeline - INFO - Finished processing product starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:15,800 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-10-06 01:05:15,801 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:05:15,978 - stpipe.Image2Pipeline - INFO - Saved model in starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits
2022-10-06 01:05:15,979 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
2022-10-06 01:05:15,984 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-10-06 01:05:15,985 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-10-06 01:05:15,987 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-10-06 01:05:15,988 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-10-06 01:05:15,989 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-10-06 01:05:15,991 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-10-06 01:05:16,145 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:16,150 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}}}
2022-10-06 01:05:16,158 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-10-06 01:05:16,161 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-10-06 01:05:16,163 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-10-06 01:05:16,164 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-10-06 01:05:16,164 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-10-06 01:05:16,164 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-10-06 01:05:16,165 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-10-06 01:05:16,166 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-10-06 01:05:16,166 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-10-06 01:05:16,167 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-10-06 01:05:16,168 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0626.fits'.
2022-10-06 01:05:16,169 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-10-06 01:05:16,169 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-10-06 01:05:16,169 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-10-06 01:05:16,170 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-10-06 01:05:16,170 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-10-06 01:05:16,170 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-10-06 01:05:16,171 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-10-06 01:05:16,171 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-10-06 01:05:16,172 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-10-06 01:05:16,172 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-10-06 01:05:16,173 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-10-06 01:05:16,173 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-10-06 01:05:16,174 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-10-06 01:05:16,174 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-10-06 01:05:16,208 - stpipe.Image2Pipeline - INFO - Processing product starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:16,209 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits> ...
2022-10-06 01:05:16,337 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:16,338 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-10-06 01:05:16,527 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-10-06 01:05:16,601 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.020403170 -0.019920319 0.022869851 0.011264898 359.991567464 0.013857704 359.988759787 -0.017341412
2022-10-06 01:05:16,602 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.020403170 -0.019920319 0.022869851 0.011264898 359.991567464 0.013857704 359.988759787 -0.017341412
2022-10-06 01:05:16,603 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-10-06 01:05:16,636 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-10-06 01:05:16,647 - stpipe.Image2Pipeline.assign_wcs - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:16,648 - stpipe.Image2Pipeline.assign_wcs - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:16,663 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-10-06 01:05:16,805 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:16,806 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-10-06 01:05:16,914 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:16,915 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:16,915 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:16,922 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:17,063 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-10-06 01:05:17,186 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:17,187 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-06 01:05:17,210 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-10-06 01:05:17,211 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-10-06 01:05:17,296 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-10-06 01:05:17,296 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-10-06 01:05:17,297 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-10-06 01:05:17,297 - stpipe.Image2Pipeline.photom - INFO - filter: F1130W
2022-10-06 01:05:17,344 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-10-06 01:05:17,346 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-10-06 01:05:17,347 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-10-06 01:05:17,348 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 2.54947
2022-10-06 01:05:17,363 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-10-06 01:05:17,487 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:17,489 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}
2022-10-06 01:05:17,515 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-10-06 01:05:17,533 - stpipe.Image2Pipeline.resample - INFO - Driz parameter kernel: square
2022-10-06 01:05:17,534 - stpipe.Image2Pipeline.resample - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:05:17,534 - stpipe.Image2Pipeline.resample - INFO - Driz parameter fillval: INDEF
2022-10-06 01:05:17,535 - stpipe.Image2Pipeline.resample - INFO - Driz parameter weight_type: ivm
2022-10-06 01:05:17,655 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-10-06 01:05:18,343 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:18,557 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-10-06 01:05:19,216 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:19,503 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-10-06 01:05:20,126 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:20,337 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-10-06 01:05:20,980 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:21,178 - stpipe.Image2Pipeline.resample - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/resample/resample.py:308: RuntimeWarning: divide by zero encountered in reciprocal
output_variance = np.reciprocal(inverse_variance_sum)
2022-10-06 01:05:21,212 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.020399927 -0.020132946 0.023150422 0.011209589 359.991531876 0.013984305 359.988781381 -0.017358229
2022-10-06 01:05:21,346 - stpipe.Image2Pipeline.resample - INFO - Saved model in starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_i2d.fits
2022-10-06 01:05:21,347 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-10-06 01:05:21,347 - stpipe.Image2Pipeline - INFO - Finished processing product starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:21,349 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-10-06 01:05:21,349 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:05:21,524 - stpipe.Image2Pipeline - INFO - Saved model in starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits
2022-10-06 01:05:21,525 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
2022-10-06 01:05:21,530 - stpipe.Image2Pipeline - INFO - Image2Pipeline instance created.
2022-10-06 01:05:21,532 - stpipe.Image2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-10-06 01:05:21,533 - stpipe.Image2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-10-06 01:05:21,534 - stpipe.Image2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-10-06 01:05:21,536 - stpipe.Image2Pipeline.photom - INFO - PhotomStep instance created.
2022-10-06 01:05:21,538 - stpipe.Image2Pipeline.resample - INFO - ResampleStep instance created.
2022-10-06 01:05:21,705 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:21,710 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': '/internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/tmp/tmpf3qlveao/starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits_cal.fits', 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_bsub': False, 'steps': {'bkg_subtract': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_combined_background': False, 'sigma': 3.0, 'maxiters': None, 'wfss_mmag_extract': None}, 'assign_wcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}, 'flat_field': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}, 'photom': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}}}
2022-10-06 01:05:21,717 - stpipe.Image2Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits' reftypes = ['area', 'camera', 'collimator', 'dflat', 'disperser', 'distortion', 'drizpars', 'fflat', 'filteroffset', 'flat', 'fore', 'fpa', 'ifufore', 'ifupost', 'ifuslicer', 'msa', 'ote', 'photom', 'regions', 'sflat', 'specwcs', 'wavelengthrange', 'wfssbkg']
2022-10-06 01:05:21,721 - stpipe.Image2Pipeline - INFO - Prefetch for AREA reference file is '/grp/crds/cache/references/jwst/jwst_miri_area_0004.fits'.
2022-10-06 01:05:21,723 - stpipe.Image2Pipeline - INFO - Prefetch for CAMERA reference file is 'N/A'.
2022-10-06 01:05:21,724 - stpipe.Image2Pipeline - INFO - Prefetch for COLLIMATOR reference file is 'N/A'.
2022-10-06 01:05:21,724 - stpipe.Image2Pipeline - INFO - Prefetch for DFLAT reference file is 'N/A'.
2022-10-06 01:05:21,725 - stpipe.Image2Pipeline - INFO - Prefetch for DISPERSER reference file is 'N/A'.
2022-10-06 01:05:21,725 - stpipe.Image2Pipeline - INFO - Prefetch for DISTORTION reference file is '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf'.
2022-10-06 01:05:21,727 - stpipe.Image2Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-10-06 01:05:21,728 - stpipe.Image2Pipeline - INFO - Prefetch for FFLAT reference file is 'N/A'.
2022-10-06 01:05:21,728 - stpipe.Image2Pipeline - INFO - Prefetch for FILTEROFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf'.
2022-10-06 01:05:21,730 - stpipe.Image2Pipeline - INFO - Prefetch for FLAT reference file is '/grp/crds/cache/references/jwst/jwst_miri_flat_0626.fits'.
2022-10-06 01:05:21,731 - stpipe.Image2Pipeline - INFO - Prefetch for FORE reference file is 'N/A'.
2022-10-06 01:05:21,732 - stpipe.Image2Pipeline - INFO - Prefetch for FPA reference file is 'N/A'.
2022-10-06 01:05:21,732 - stpipe.Image2Pipeline - INFO - Prefetch for IFUFORE reference file is 'N/A'.
2022-10-06 01:05:21,733 - stpipe.Image2Pipeline - INFO - Prefetch for IFUPOST reference file is 'N/A'.
2022-10-06 01:05:21,733 - stpipe.Image2Pipeline - INFO - Prefetch for IFUSLICER reference file is 'N/A'.
2022-10-06 01:05:21,734 - stpipe.Image2Pipeline - INFO - Prefetch for MSA reference file is 'N/A'.
2022-10-06 01:05:21,734 - stpipe.Image2Pipeline - INFO - Prefetch for OTE reference file is 'N/A'.
2022-10-06 01:05:21,734 - stpipe.Image2Pipeline - INFO - Prefetch for PHOTOM reference file is '/grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits'.
2022-10-06 01:05:21,736 - stpipe.Image2Pipeline - INFO - Prefetch for REGIONS reference file is 'N/A'.
2022-10-06 01:05:21,736 - stpipe.Image2Pipeline - INFO - Prefetch for SFLAT reference file is 'N/A'.
2022-10-06 01:05:21,736 - stpipe.Image2Pipeline - INFO - Prefetch for SPECWCS reference file is 'N/A'.
2022-10-06 01:05:21,737 - stpipe.Image2Pipeline - INFO - Prefetch for WAVELENGTHRANGE reference file is 'N/A'.
2022-10-06 01:05:21,737 - stpipe.Image2Pipeline - INFO - Prefetch for WFSSBKG reference file is 'N/A'.
2022-10-06 01:05:21,737 - stpipe.Image2Pipeline - INFO - Starting calwebb_image2 ...
2022-10-06 01:05:21,772 - stpipe.Image2Pipeline - INFO - Processing product starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:21,773 - stpipe.Image2Pipeline - INFO - Working on input <ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits> ...
2022-10-06 01:05:21,916 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:21,917 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'sip_approx': True, 'sip_max_pix_error': 0.25, 'sip_degree': None, 'sip_max_inv_pix_error': 0.25, 'sip_inv_degree': None, 'sip_npoints': 32, 'slit_y_low': -0.55, 'slit_y_high': 0.55}
2022-10-06 01:05:22,115 - stpipe.Image2Pipeline.assign_wcs - INFO - Created a MIRI mir_image pipeline with references {'distortion': '/grp/crds/cache/references/jwst/jwst_miri_distortion_0028.asdf', 'filteroffset': '/grp/crds/cache/references/jwst/jwst_miri_filteroffset_0005.asdf', 'specwcs': None, 'regions': None, 'wavelengthrange': None, 'camera': None, 'collimator': None, 'disperser': None, 'fore': None, 'fpa': None, 'msa': None, 'ote': None, 'ifupost': None, 'ifufore': None, 'ifuslicer': None}
2022-10-06 01:05:22,193 - stpipe.Image2Pipeline.assign_wcs - INFO - Update S_REGION to POLYGON ICRS 0.021939052 -0.019421471 0.024405733 0.011763747 359.993103346 0.014356552 359.990295669 -0.016842564
2022-10-06 01:05:22,194 - stpipe.Image2Pipeline.assign_wcs - INFO - assign_wcs updated S_REGION to POLYGON ICRS 0.021939052 -0.019421471 0.024405733 0.011763747 359.993103346 0.014356552 359.990295669 -0.016842564
2022-10-06 01:05:22,194 - stpipe.Image2Pipeline.assign_wcs - INFO - COMPLETED assign_wcs
2022-10-06 01:05:22,230 - stpipe.Image2Pipeline.assign_wcs - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-10-06 01:05:22,242 - stpipe.Image2Pipeline.assign_wcs - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:22,242 - stpipe.Image2Pipeline.assign_wcs - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:22,257 - stpipe.Image2Pipeline.assign_wcs - INFO - Step assign_wcs done
2022-10-06 01:05:22,416 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:22,417 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_interpolated_flat': False, 'user_supplied_flat': None, 'inverse': False}
2022-10-06 01:05:22,532 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_PARTIAL_DATA does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:22,533 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_LOW_QUAL does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:22,533 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword CDP_UNRELIABLE_ERROR does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:22,541 - stpipe.Image2Pipeline.flat_field - WARNING - Keyword DIFF_PATTERN does not correspond to an existing DQ mnemonic, so will be ignored
2022-10-06 01:05:22,675 - stpipe.Image2Pipeline.flat_field - INFO - Step flat_field done
2022-10-06 01:05:22,818 - stpipe.Image2Pipeline.photom - INFO - Step photom running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:22,820 - stpipe.Image2Pipeline.photom - INFO - Step photom parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'inverse': False, 'source_type': None}
2022-10-06 01:05:22,845 - stpipe.Image2Pipeline.photom - INFO - Using photom reference file: /grp/crds/cache/references/jwst/jwst_miri_photom_0073.fits
2022-10-06 01:05:22,846 - stpipe.Image2Pipeline.photom - INFO - Using area reference file: /grp/crds/cache/references/jwst/jwst_miri_area_0004.fits
2022-10-06 01:05:22,934 - stpipe.Image2Pipeline.photom - INFO - Using instrument: MIRI
2022-10-06 01:05:22,935 - stpipe.Image2Pipeline.photom - INFO - detector: MIRIMAGE
2022-10-06 01:05:22,936 - stpipe.Image2Pipeline.photom - INFO - exp_type: MIR_IMAGE
2022-10-06 01:05:22,936 - stpipe.Image2Pipeline.photom - INFO - filter: F1130W
2022-10-06 01:05:22,985 - stpipe.Image2Pipeline.photom - INFO - Pixel area map copied to output.
2022-10-06 01:05:22,988 - stpipe.Image2Pipeline.photom - INFO - subarray: FULL
2022-10-06 01:05:22,988 - stpipe.Image2Pipeline.photom - WARNING - Expected to find one matching row in table, found 0.
2022-10-06 01:05:22,989 - stpipe.Image2Pipeline.photom - INFO - PHOTMJSR value: 2.54947
2022-10-06 01:05:23,004 - stpipe.Image2Pipeline.photom - INFO - Step photom done
2022-10-06 01:05:23,139 - stpipe.Image2Pipeline.resample - INFO - Step resample running with args (<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits>,).
2022-10-06 01:05:23,140 - stpipe.Image2Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}
2022-10-06 01:05:23,167 - stpipe.Image2Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-10-06 01:05:23,186 - stpipe.Image2Pipeline.resample - INFO - Driz parameter kernel: square
2022-10-06 01:05:23,187 - stpipe.Image2Pipeline.resample - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:05:23,188 - stpipe.Image2Pipeline.resample - INFO - Driz parameter fillval: INDEF
2022-10-06 01:05:23,188 - stpipe.Image2Pipeline.resample - INFO - Driz parameter weight_type: ivm
2022-10-06 01:05:23,309 - stpipe.Image2Pipeline.resample - INFO - Resampling science data
2022-10-06 01:05:23,907 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:24,094 - stpipe.Image2Pipeline.resample - INFO - Resampling var_rnoise
2022-10-06 01:05:24,681 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:24,879 - stpipe.Image2Pipeline.resample - INFO - Resampling var_poisson
2022-10-06 01:05:25,449 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:25,653 - stpipe.Image2Pipeline.resample - INFO - Resampling var_flat
2022-10-06 01:05:26,231 - stpipe.Image2Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1022, 1031)
2022-10-06 01:05:26,432 - stpipe.Image2Pipeline.resample - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/resample/resample.py:308: RuntimeWarning: divide by zero encountered in reciprocal
output_variance = np.reciprocal(inverse_variance_sum)
2022-10-06 01:05:26,467 - stpipe.Image2Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021935809 -0.019634097 0.024686304 0.011708437 359.993067758 0.014483153 359.990317263 -0.016859381
2022-10-06 01:05:26,603 - stpipe.Image2Pipeline.resample - INFO - Saved model in starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_i2d.fits
2022-10-06 01:05:26,604 - stpipe.Image2Pipeline.resample - INFO - Step resample done
2022-10-06 01:05:26,604 - stpipe.Image2Pipeline - INFO - Finished processing product starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits_cal
2022-10-06 01:05:26,605 - stpipe.Image2Pipeline - INFO - ... ending calwebb_image2
2022-10-06 01:05:26,606 - stpipe.Image2Pipeline - INFO - Results used CRDS context: jwst_0988.pmap
2022-10-06 01:05:26,766 - stpipe.Image2Pipeline - INFO - Saved model in starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits
2022-10-06 01:05:26,767 - stpipe.Image2Pipeline - INFO - Step Image2Pipeline done
[[<ImageModel(1024, 1032) from starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits>], [<ImageModel(1024, 1032) from starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits>], [<ImageModel(1024, 1032) from starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits>], [<ImageModel(1024, 1032) from starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits>]]
# Get an ordered list of the calibrated file names
print(ratefilenames)
cal_files = [ele.replace('rate', 'cal') for ele in ratefilenames]
print(cal_files)
['starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_rate.fits', 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_rate.fits', 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_rate.fits', 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_rate.fits'] ['starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits']
The calwebb_image3 pipeline takes in an association table of a set of images which should be combined. The association table can also be used to specify a background image to be subtracted or a source catalog to be used within the pipeline (sourcecat is not typically used with MIRI).
# use asn_from_list to create association table
asn = asn_from_list.asn_from_list(cal_files, rule=DMS_Level3_Base, product_name='starfield_50star4ptdither_combined.fits')
# use this if you need to add non'science' exposure types
#asn['products'][0]['members'][1]['exptype'] = 'background'
#asn['products'][0]['members'][2]['exptype'] = 'sourcecat'
# dump association table to a .json file for use in image3
with open('starfield_50star4ptdither_asnfile.json', 'w') as fp:
fp.write(asn.dump()[1])
print(asn)
print(cal_files)
jwnoprogram-a3001_none_006_asn with 1 products Rule=DMS_Level3_Base No constraints Products: starfield_50star4ptdither_combined.fits with 4 members ['starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits']
Insert 'outliers' into the files to see if they are detected by the outlier_detection step.
pixloc = []
for i in range(len(cal_files)):
pixloc.append([random.randint(20,1010),random.randint(430,1010)])
pixloc.append([random.randint(20,1010),random.randint(430,1010)])
pixloc.append([random.randint(20,1010),random.randint(430,1010)])
pixloc.append([random.randint(20,1010),random.randint(430,1010)])
pixloc2 = np.array(pixloc)
print(pixloc2)
[[542 705] [598 969] [504 871] [482 693] [403 633] [527 692] [507 550] [ 59 600] [701 825] [600 940] [870 627] [134 479] [799 481] [233 479] [565 671] [948 526]]
# This block of code caused a pipeline crash in image3. Use the cell below instead, as that ran to completion
#for i in range(len(cal_files)):
# with ImageModel(cal_files[i]) as im:
# print(im.meta.filename)
# j = 4*i
# med = np.median(im.data)
# print(med)
# Put in elevated flux values at randomized coordinates
# im.data[pixloc2[j,0],pixloc2[j,1]] = med*5.0
# im.data[pixloc2[j+1,0],pixloc2[j+1,1]] = med*10.0
# im.data[pixloc2[j+2,0],pixloc2[j+2,1]] = med*15.0
# im.data[pixloc2[j+3,0],pixloc2[j+3,1]] = med*20.0
# im.save(im.meta.filename)
for i in range(len(cal_files)):
with fits.open(cal_files[i]) as h:
j = 4*i
med = np.median(h['SCI'].data)
h['SCI'].data[pixloc2[j,0],pixloc2[j,1]] = med*5.0
h['SCI'].data[pixloc2[j+1,0],pixloc2[j+1,1]] = med*10.0
h['SCI'].data[pixloc2[j+2,0],pixloc2[j+2,1]] = med*15.0
h['SCI'].data[pixloc2[j+3,0],pixloc2[j+3,1]] = med*20.0
h.writeto(cal_files[i],overwrite=True)
# Look at the images with outliers added to see if any landed on sources or might be less likely to be detected
i = 0
for image in cal_files:
imagetest = ImageModel(image)
print(imagetest.meta.filename)
print('Min value ', np.nanmin(imagetest.data), 'Max value ', np.nanmax(imagetest.data))
j = 4*i
crs = pixloc2[j:j+4]
print(crs)
yvals = pixloc2[j:j+4,0]
xvals = pixloc2[j:j+4,1]
plt.figure(figsize=(15,15))
plt.imshow(imagetest.data,cmap="Greys",origin='lower',vmin=0, vmax=50)
plt.colorbar()
plt.scatter(xvals, yvals,lw=1, s=10,color='red')
i += 1
starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits Min value -140.43025 Max value 5297.299 [[542 705] [598 969] [504 871] [482 693]] starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits Min value -140.15489 Max value 5164.367 [[403 633] [527 692] [507 550] [ 59 600]] starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits Min value -141.17201 Max value 5308.2524 [[701 825] [600 940] [870 627] [134 479]] starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits Min value -140.16364 Max value 5474.6606 [[799 481] [233 479] [565 671] [948 526]]
# use association table created in previous step with calwebb_image3
# set any specific parameters
# tweakreg parameters to allow data to run
fwhm=3.762 # Gaussian kernel FWHM of objects expected, default=2.5
minobj=5 # minimum number of objects needed to match positions for a good fit, default=15
snr= 100 # signal to noise threshold, default=5
sigma= 3 # clipping limit, in sigma units, used when performing fit, default=3
fit_geom='shift' # ftype of affine transformation to be considered when fitting catalogs, default='general'
use2dhist=False # boolean indicating whether to use 2D histogram to find initial offset, default=True
pipe3=Image3Pipeline()
pipe3.tweakreg.kernel_fwhm = fwhm
pipe3.tweakreg.snr_threshold = snr
pipe3.tweakreg.minobj = minobj
pipe3.tweakreg.sigma = sigma
pipe3.tweakreg.fitgeometry = fit_geom
pipe3.tweakreg.use2dhist = use2dhist
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.source_catalog.snr_threshold = snr
pipe3.skymatch.save_results = True
pipe3.outlier_detection.save_results = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.save_results = True
# run Image3
pipe3.run('starfield_50star4ptdither_asnfile.json')
print('Image 3 pipeline finished.')
2022-10-06 01:05:29,891 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-10-06 01:05:29,892 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-10-06 01:05:29,894 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-10-06 01:05:29,895 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-10-06 01:05:29,897 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-10-06 01:05:29,898 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-10-06 01:05:29,899 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-10-06 01:05:30,057 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('starfield_50star4ptdither_asnfile.json',).
2022-10-06 01:05:30,064 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'sharplo': 0.2, 'sharphi': 1.0, 'roundlo': -1.0, 'roundhi': 1.0, 'brightest': 200, 'peakmax': None, 'bkg_boxsize': 400, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 5, 'searchrad': 2.0, 'use2dhist': False, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'shift', 'nclip': 3, 'sigma': 3, 'abs_refcat': '', 'save_abs_catalog': False, 'abs_minobj': 15, 'abs_searchrad': 6.0, 'abs_use2dhist': True, 'abs_separation': 0.1, 'abs_tolerance': 0.7, 'abs_fitgeometry': 'rshift', 'abs_nclip': 3, 'abs_sigma': 3.0}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None, 'in_memory': False}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 1000, 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'npixels': 25, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-10-06 01:05:30,197 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-10-06 01:05:30,214 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-10-06 01:05:30,215 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-10-06 01:05:30,217 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-10-06 01:05:30,218 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-10-06 01:05:30,838 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-10-06 01:05:30,840 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'sharplo': 0.2, 'sharphi': 1.0, 'roundlo': -1.0, 'roundhi': 1.0, 'brightest': 200, 'peakmax': None, 'bkg_boxsize': 400, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 5, 'searchrad': 2.0, 'use2dhist': False, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'shift', 'nclip': 3, 'sigma': 3, 'abs_refcat': '', 'save_abs_catalog': False, 'abs_minobj': 15, 'abs_searchrad': 6.0, 'abs_use2dhist': True, 'abs_separation': 0.1, 'abs_tolerance': 0.7, 'abs_fitgeometry': 'rshift', 'abs_nclip': 3, 'abs_sigma': 3.0}
2022-10-06 01:05:31,009 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 200 sources in starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:05:31,155 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 200 sources in starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:05:31,299 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 183 sources in starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:05:31,443 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 179 sources in starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:05:31,463 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:31,464 - stpipe.Image3Pipeline.tweakreg - INFO - Number of image groups to be aligned: 4.
2022-10-06 01:05:31,464 - stpipe.Image3Pipeline.tweakreg - INFO - Image groups:
2022-10-06 01:05:31,490 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:05:31,490 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:05:31,516 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:05:31,516 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:05:31,541 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:05:31,542 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:05:31,566 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:05:31,567 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:05:31,567 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:31,567 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:31,568 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() started on 2022-10-06 01:05:31.567916
2022-10-06 01:05:31,568 - stpipe.Image3Pipeline.tweakreg - INFO - Version 0.8.0
2022-10-06 01:05:31,568 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:31,700 - stpipe.Image3Pipeline.tweakreg - INFO - Selected image 'GROUP ID: starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' as reference image
2022-10-06 01:05:31,705 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal' to the reference catalog.
2022-10-06 01:05:31,828 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal' catalog with sources from the reference 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' catalog.
2022-10-06 01:05:31,829 - stpipe.Image3Pipeline.tweakreg - INFO - Found 91 matches for 'GROUP ID: starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal'...
2022-10-06 01:05:31,830 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'shift' fit
2022-10-06 01:05:31,832 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'shift' fit for GROUP ID: starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal:
2022-10-06 01:05:31,832 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: 0.0187591 YSH: 0.00410462
2022-10-06 01:05:31,833 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:31,833 - stpipe.Image3Pipeline.tweakreg - INFO - FIT RMSE: 0.32968 FIT MAE: 0.182423
2022-10-06 01:05:31,833 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 91 objects.
2022-10-06 01:05:31,872 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal' to the reference catalog.
2022-10-06 01:05:32,000 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal' catalog with sources from the reference 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' catalog.
2022-10-06 01:05:32,001 - stpipe.Image3Pipeline.tweakreg - INFO - Found 79 matches for 'GROUP ID: starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal'...
2022-10-06 01:05:32,002 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'shift' fit
2022-10-06 01:05:32,004 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'shift' fit for GROUP ID: starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal:
2022-10-06 01:05:32,005 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: -0.0039417 YSH: 0.00507471
2022-10-06 01:05:32,005 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:32,006 - stpipe.Image3Pipeline.tweakreg - INFO - FIT RMSE: 0.323486 FIT MAE: 0.171732
2022-10-06 01:05:32,006 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 79 objects.
2022-10-06 01:05:32,045 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal' to the reference catalog.
2022-10-06 01:05:32,174 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal' catalog with sources from the reference 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' catalog.
2022-10-06 01:05:32,175 - stpipe.Image3Pipeline.tweakreg - INFO - Found 74 matches for 'GROUP ID: starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal'...
2022-10-06 01:05:32,176 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'shift' fit
2022-10-06 01:05:32,178 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'shift' fit for GROUP ID: starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal:
2022-10-06 01:05:32,179 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: -0.0599251 YSH: -0.0277362
2022-10-06 01:05:32,179 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:32,179 - stpipe.Image3Pipeline.tweakreg - INFO - FIT RMSE: 0.3174 FIT MAE: 0.203801
2022-10-06 01:05:32,180 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 74 objects.
2022-10-06 01:05:32,220 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:32,220 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() ended on 2022-10-06 01:05:32.220201
2022-10-06 01:05:32,221 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() TOTAL RUN TIME: 0:00:00.652285
2022-10-06 01:05:32,221 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:05:32,357 - stpipe.Image3Pipeline.tweakreg - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-10-06 01:05:32,369 - stpipe.Image3Pipeline.tweakreg - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:32,370 - stpipe.Image3Pipeline.tweakreg - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:32,485 - stpipe.Image3Pipeline.tweakreg - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:32,486 - stpipe.Image3Pipeline.tweakreg - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:32,603 - stpipe.Image3Pipeline.tweakreg - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:05:32,604 - stpipe.Image3Pipeline.tweakreg - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:05:32,611 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-10-06 01:05:32,798 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-10-06 01:05:32,800 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-10-06 01:05:32,920 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:05:32,921 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-10-06 01:05:32.920044
2022-10-06 01:05:32,921 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:05:32,922 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'match'
2022-10-06 01:05:32,923 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN
2022-10-06 01:05:32,923 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF
2022-10-06 01:05:32,924 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:05:32,924 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-10-06 01:05:35,525 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0.0162815
2022-10-06 01:05:35,526 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0.0246113
2022-10-06 01:05:35,527 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0.0157997
2022-10-06 01:05:35,527 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0
2022-10-06 01:05:35,527 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:05:35,528 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-10-06 01:05:35.527784
2022-10-06 01:05:35,528 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.607740
2022-10-06 01:05:35,529 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:05:35,805 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_0_skymatch.fits
2022-10-06 01:05:36,092 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_1_skymatch.fits
2022-10-06 01:05:36,386 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_2_skymatch.fits
2022-10-06 01:05:36,690 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_3_skymatch.fits
2022-10-06 01:05:36,691 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-10-06 01:05:36,861 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-10-06 01:05:36,863 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None, 'in_memory': False}
2022-10-06 01:05:36,870 - stpipe.Image3Pipeline.outlier_detection - INFO - Performing outlier detection on 4 inputs
2022-10-06 01:05:36,870 - stpipe.Image3Pipeline.outlier_detection - INFO - Driz parameter kernel: square
2022-10-06 01:05:36,871 - stpipe.Image3Pipeline.outlier_detection - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:05:36,871 - stpipe.Image3Pipeline.outlier_detection - INFO - Driz parameter fillval: INDEF
2022-10-06 01:05:36,871 - stpipe.Image3Pipeline.outlier_detection - INFO - Driz parameter weight_type: ivm
2022-10-06 01:05:37,008 - stpipe.Image3Pipeline.outlier_detection - INFO - 1 exposures to drizzle together
2022-10-06 01:05:37,621 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:37,904 - stpipe.Image3Pipeline.outlier_detection - INFO - Exposure starfield_50star4ptdither_combined_0_outlier_i2d.fits saved to file
2022-10-06 01:05:37,908 - stpipe.Image3Pipeline.outlier_detection - INFO - 1 exposures to drizzle together
2022-10-06 01:05:38,728 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:39,018 - stpipe.Image3Pipeline.outlier_detection - INFO - Exposure starfield_50star4ptdither_combined_1_outlier_i2d.fits saved to file
2022-10-06 01:05:39,023 - stpipe.Image3Pipeline.outlier_detection - INFO - 1 exposures to drizzle together
2022-10-06 01:05:39,840 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:40,127 - stpipe.Image3Pipeline.outlier_detection - INFO - Exposure starfield_50star4ptdither_combined_2_outlier_i2d.fits saved to file
2022-10-06 01:05:40,132 - stpipe.Image3Pipeline.outlier_detection - INFO - 1 exposures to drizzle together
2022-10-06 01:05:40,942 - stpipe.Image3Pipeline.outlier_detection - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:41,243 - stpipe.Image3Pipeline.outlier_detection - INFO - Exposure starfield_50star4ptdither_combined_3_outlier_i2d.fits saved to file
2022-10-06 01:05:44,398 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting median...
2022-10-06 01:05:44,496 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:335: RuntimeWarning: invalid value encountered in multiply
blotted_median.err *= 0.0 # None
2022-10-06 01:05:45,117 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1118)
2022-10-06 01:05:45,434 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:335: RuntimeWarning: invalid value encountered in multiply
blotted_median.err *= 0.0 # None
2022-10-06 01:05:46,217 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1118)
2022-10-06 01:05:46,552 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:335: RuntimeWarning: invalid value encountered in multiply
blotted_median.err *= 0.0 # None
2022-10-06 01:05:47,337 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1118)
2022-10-06 01:05:47,675 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:335: RuntimeWarning: invalid value encountered in multiply
blotted_median.err *= 0.0 # None
2022-10-06 01:05:48,466 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (1024, 1032) <-- (1142, 1118)
2022-10-06 01:05:48,690 - stpipe.Image3Pipeline.outlier_detection - INFO - Flagging outliers
2022-10-06 01:05:49,078 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:440: RuntimeWarning: overflow encountered in multiply
threshold1 = scale1 * blot_deriv + snr1 * err_data
2022-10-06 01:05:49,099 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:449: RuntimeWarning: overflow encountered in multiply
threshold2 = scale2 * blot_deriv + snr2 * err_data
2022-10-06 01:05:49,114 - stpipe.Image3Pipeline.outlier_detection - INFO - New pixels flagged as outliers: 459 (0.04%)
2022-10-06 01:05:49,316 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:440: RuntimeWarning: overflow encountered in multiply
threshold1 = scale1 * blot_deriv + snr1 * err_data
2022-10-06 01:05:49,336 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:449: RuntimeWarning: overflow encountered in multiply
threshold2 = scale2 * blot_deriv + snr2 * err_data
2022-10-06 01:05:49,350 - stpipe.Image3Pipeline.outlier_detection - INFO - New pixels flagged as outliers: 548 (0.05%)
2022-10-06 01:05:49,554 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:440: RuntimeWarning: overflow encountered in multiply
threshold1 = scale1 * blot_deriv + snr1 * err_data
2022-10-06 01:05:49,574 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:449: RuntimeWarning: overflow encountered in multiply
threshold2 = scale2 * blot_deriv + snr2 * err_data
2022-10-06 01:05:49,587 - stpipe.Image3Pipeline.outlier_detection - INFO - New pixels flagged as outliers: 384 (0.04%)
2022-10-06 01:05:49,790 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:440: RuntimeWarning: overflow encountered in multiply
threshold1 = scale1 * blot_deriv + snr1 * err_data
2022-10-06 01:05:49,810 - stpipe.Image3Pipeline.outlier_detection - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/outlier_detection/outlier_detection.py:449: RuntimeWarning: overflow encountered in multiply
threshold2 = scale2 * blot_deriv + snr2 * err_data
2022-10-06 01:05:49,823 - stpipe.Image3Pipeline.outlier_detection - INFO - New pixels flagged as outliers: 1338 (0.13%)
2022-10-06 01:05:50,083 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in starfield_50star4ptdither_combined_0_a3001_crf.fits
2022-10-06 01:05:50,367 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in starfield_50star4ptdither_combined_1_a3001_crf.fits
2022-10-06 01:05:50,653 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in starfield_50star4ptdither_combined_2_a3001_crf.fits
2022-10-06 01:05:50,932 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in starfield_50star4ptdither_combined_3_a3001_crf.fits
2022-10-06 01:05:50,933 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-10-06 01:05:51,115 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-10-06 01:05:51,117 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}
2022-10-06 01:05:51,134 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-10-06 01:05:51,154 - stpipe.Image3Pipeline.resample - INFO - Driz parameter kernel: square
2022-10-06 01:05:51,155 - stpipe.Image3Pipeline.resample - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:05:51,155 - stpipe.Image3Pipeline.resample - INFO - Driz parameter fillval: INDEF
2022-10-06 01:05:51,156 - stpipe.Image3Pipeline.resample - INFO - Driz parameter weight_type: ivm
2022-10-06 01:05:51,306 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for starfield_50star4ptdither_combined.fits
2022-10-06 01:05:51,956 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-10-06 01:05:52,543 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:53,509 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:54,590 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:55,557 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:55,765 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-10-06 01:05:56,369 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:57,339 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:58,298 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:59,316 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:05:59,514 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-10-06 01:06:00,119 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:01,160 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:02,178 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:03,193 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:03,387 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-10-06 01:06:03,993 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:05,020 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:06,139 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:07,127 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:07,331 - stpipe.Image3Pipeline.resample - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/resample/resample.py:308: RuntimeWarning: divide by zero encountered in reciprocal
output_variance = np.reciprocal(inverse_variance_sum)
2022-10-06 01:06:07,369 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021864896 -0.020260192 0.024938345 0.014762483 359.990651697 0.017771342 359.987578248 -0.017251333
2022-10-06 01:06:07,833 - stpipe.Image3Pipeline.resample - INFO - Saved model in starfield_50star4ptdither_combined_i2d.fits
2022-10-06 01:06:07,833 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-10-06 01:06:08,040 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from starfield_50star4ptdither_combined_i2d.fits>,).
2022-10-06 01:06:08,042 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 1000, 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'npixels': 25, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-10-06 01:06:08,064 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-10-06 01:06:08,074 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-10-06 01:06:08,075 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-10-06 01:06:08,075 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-10-06 01:06:08,076 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F1130W
2022-10-06 01:06:08,076 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-10-06 01:06:08,152 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 5.49349
2022-10-06 01:06:08,230 - stpipe.Image3Pipeline.source_catalog - INFO - Background could not be estimated in meshes. Using the entire unmasked array for background estimation: bkg_boxsize=(1142, 1118).
2022-10-06 01:06:08,806 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 46 sources
2022-10-06 01:06:08,951 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: starfield_50star4ptdither_combined_cat.ecsv
2022-10-06 01:06:09,092 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in starfield_50star4ptdither_combined_segm.fits
2022-10-06 01:06:09,094 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: starfield_50star4ptdither_combined_segm.fits
2022-10-06 01:06:09,095 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-10-06 01:06:09,096 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
View the image and check number of pixels being flagged as outliers.
# read in i2d file
im_i2d = ImageModel('starfield_50star4ptdither_combined_i2d.fits')
#norm = ImageNormalize(im_i2d.data, interval=MinMaxInterval(),
# stretch=LinearStretch())
# mask out DO_NOT_USE values where data has been set to 0 in the combined image
masked_im = np.ma.masked_where((im_i2d.data == 0), im_i2d.data)
cmap = matplotlib.cm.get_cmap("Greys").copy() # Can be any colormap that you want after the cm
cmap.set_bad(color='white') # color to mark all DO_NOT_USE pixels
plt.figure(figsize=(15,15))
#plt.imshow(im_i2d.data,origin='lower',norm=norm,vmin=-5, vmax=4)
plt.imshow(masked_im, origin='lower', cmap = cmap, vmin=5, vmax=20)
plt.colorbar()
<matplotlib.colorbar.Colorbar at 0x7f93b48c3340>
The image output should have the Four Quadrant phase masks on the left of the image masked out (values of 0). The image area should be smooth in the background regions with multiple point sources bright against the background. Passing criteria for the Lyot mask region are still being determined.
photfile = 'starfield_50star4ptdither_combined_cat.ecsv'
data = table.Table.read(photfile, format='ascii', comment='#')
print(len(data),' sources detected')
46 sources detected
# read in ecsv photom file
from astropy.visualization import LogStretch, PercentileInterval, ManualInterval
from astropy import table
from matplotlib.colors import LogNorm
from astropy.visualization import (MinMaxInterval, SqrtStretch,
ImageNormalize)
norm = ImageNormalize(im_i2d.data, interval=MinMaxInterval(),
stretch=SqrtStretch())
plt.figure(figsize=(15,15))
#plt.imshow(viz2(im_i2d.data),cmap='gray')
plt.imshow(im_i2d.data,origin='lower', vmin=5, vmax=15)
plt.colorbar()
plt.scatter(data['xcentroid'], data['ycentroid'],lw=1, s=10,color='red')
<matplotlib.collections.PathCollection at 0x7f93b8b1df40>
If the red dots marking sources found in image above are centered on the point sources, the test passes.
Read in the individual crf files which are output from outlier_detection and check the dq extension to see how many pixels out of each image are flagged as outliers.
import glob
image3crffiles = glob.glob('*crf.fits')
output_files = sorted(image3crffiles)
print(image3crffiles)
print()
print(output_files)
['starfield_50star4ptdither_combined_0_a3001_crf.fits', 'starfield_50star4ptdither_combined_1_a3001_crf.fits', 'starfield_50star4ptdither_combined_2_a3001_crf.fits', 'starfield_50star4ptdither_combined_3_a3001_crf.fits'] ['starfield_50star4ptdither_combined_0_a3001_crf.fits', 'starfield_50star4ptdither_combined_1_a3001_crf.fits', 'starfield_50star4ptdither_combined_2_a3001_crf.fits', 'starfield_50star4ptdither_combined_3_a3001_crf.fits']
# Get the names of the crf files as output by calwebb_image3.
# This handwritten list is based on the naming of the files as they appear at the time the code was written (May 2021).
# If the filename convention changes, this block will need to be updated.
#output_files = []
#counter = 0
#for item in cal_files:
# name = 'starfield_50star4ptdither_combined'
# crffile = name+"_"+str(counter)+"_a3001_crf.fits"
# output_files.append(crffile)
# counter +=1
# Print calfiles and output_files to see the mapping of the filenames
print(cal_files)
print()
print(output_files)
['starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits', 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits'] ['starfield_50star4ptdither_combined_0_a3001_crf.fits', 'starfield_50star4ptdither_combined_1_a3001_crf.fits', 'starfield_50star4ptdither_combined_2_a3001_crf.fits', 'starfield_50star4ptdither_combined_3_a3001_crf.fits']
dq_after = []
all_out_dqs = []
for i in range(len(output_files)):
with ImageModel(output_files[i]) as crf:
j = 4*i
dq_after.append([pixloc2[j,:],crf.dq[pixloc2[j,0],pixloc2[j,1]]])
dq_after.append([pixloc2[j+1,:],crf.dq[pixloc2[j+1,0],pixloc2[j+1,1]]])
dq_after.append([pixloc2[j+2,:],crf.dq[pixloc2[j+2,0],pixloc2[j+2,1]]])
dq_after.append([pixloc2[j+3,:],crf.dq[pixloc2[j+3,0],pixloc2[j+3,1]]])
all_out_dqs.append((crf.dq[pixloc2[j,0],pixloc2[j,1]] & dqflags.pixel['OUTLIER'] > 0))
all_out_dqs.append((crf.dq[pixloc2[j+1,0],pixloc2[j+1,1]] & dqflags.pixel['OUTLIER'] > 0))
all_out_dqs.append((crf.dq[pixloc2[j+2,0],pixloc2[j+2,1]] & dqflags.pixel['OUTLIER'] > 0))
all_out_dqs.append((crf.dq[pixloc2[j+3,0],pixloc2[j+3,1]] & dqflags.pixel['OUTLIER'] > 0))
print(dq_after)
print()
print(all_out_dqs)
[[array([542, 705]), 17], [array([598, 969]), 17], [array([504, 871]), 17], [array([482, 693]), 17], [array([403, 633]), 17], [array([527, 692]), 17], [array([507, 550]), 17], [array([ 59, 600]), 17], [array([701, 825]), 17], [array([600, 940]), 17], [array([870, 627]), 17], [array([134, 479]), 0], [array([799, 481]), 17], [array([233, 479]), 17], [array([565, 671]), 17], [array([948, 526]), 17]] [True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True]
# Read in each crf file output from outlier_detection and see percentage of pixels flagged as outlier
flag_thresh = 1.0 # Percentage above which user should be notified of high percentage of flagged pixels
for crffile in output_files:
file = ImageModel(crffile)
nx = file.meta.subarray.xsize
ny = file.meta.subarray.ysize
filename = file.meta.filename
print(filename)
numpix = nx * ny
# Test that all pixels flagged with OUTLIER are also flagged as DO_NOT_USE
outlierarray = (file.dq & dqflags.pixel['OUTLIER'] > 0)
badarray = (file.dq & dqflags.pixel['DO_NOT_USE'] > 0)
try:
assert outlierarray.all() == badarray.all()
except:
print('Pixels flagged as outliers and "DO_NOT_USE" do not match')
# Count number of pixels flagged as OUTLIER
jumpcount = (file.dq & dqflags.pixel['OUTLIER'] > 0).sum()
print('There are ', jumpcount, ' pixels flagged as outliers.')
percentflagged = (jumpcount / numpix) * 100.
print('The percentage of pixels flagged is ', percentflagged)
if percentflagged > flag_thresh:
print('This percentage is higher than it should be. Review data through outlier step')
print('\n')
starfield_50star4ptdither_combined_0_a3001_crf.fits There are 2181 pixels flagged as outliers. The percentage of pixels flagged is 0.20638399345930233 starfield_50star4ptdither_combined_1_a3001_crf.fits There are 3652 pixels flagged as outliers. The percentage of pixels flagged is 0.3455820009689923 starfield_50star4ptdither_combined_2_a3001_crf.fits There are 4245 pixels flagged as outliers. The percentage of pixels flagged is 0.4016964934593023 starfield_50star4ptdither_combined_3_a3001_crf.fits There are 5510 pixels flagged as outliers. The percentage of pixels flagged is 0.5214011022286822
# Print out whether all of the added cr hits were flagged as outliers
print('Output DQ values: ', all_out_dqs)
try:
assert np.alltrue(all_out_dqs) == True
print('MIRI Outlier Detection test: Passed')
except:
print('AssertionError: At least one cr hit was not flagged')
Output DQ values: [True, True, True, True, True, True, True, True, True, True, True, False, True, True, True, True] AssertionError: At least one cr hit was not flagged
The output above should show percentages below some chosen threshold. If the percentage of pixels flagged as outliers are above the set threshold, there will be error messages printed. If no error messages are printed, this test is presumed to pass.
asn = asn_from_list.asn_from_list(cal_files, rule=DMS_Level3_Base, product_name='starfield_50star4ptdither_combined_no_outlier.fits')
# use this if you need to add non'science' exposure types
#asn['products'][0]['members'][1]['exptype'] = 'background'
#asn['products'][0]['members'][2]['exptype'] = 'sourcecat'
# dump association table to a .json file for use in image3
with open('starfield_50star4ptdither_no_outlier_asnfile.json', 'w') as fp:
fp.write(asn.dump()[1])
# use association table created in previous step with calwebb_image3
# set any specific parameters
# tweakreg parameters to allow data to run
fwhm=3.762 # Gaussian kernel FWHM of objects expected, default=2.5
minobj=5 # minimum number of objects needed to match positions for a good fit, default=15
snr= 100 # signal to noise threshold, default=5
sigma= 3 # clipping limit, in sigma units, used when performing fit, default=3
fit_geom='shift' # ftype of affine transformation to be considered when fitting catalogs, default='general'
use2dhist=False # boolean indicating whether to use 2D histogram to find initial offset, default=True
pipe3=Image3Pipeline()
pipe3.tweakreg.kernel_fwhm = fwhm
pipe3.tweakreg.snr_threshold = snr
pipe3.tweakreg.minobj = minobj
pipe3.tweakreg.sigma = sigma
pipe3.tweakreg.fitgeometry = fit_geom
pipe3.tweakreg.use2dhist = use2dhist
pipe3.source_catalog.kernel_fwhm = fwhm
pipe3.source_catalog.snr_threshold = snr
pipe3.skymatch.save_results = True
pipe3.outlier_detection.skip = True
pipe3.resample.save_results = True
pipe3.source_catalog.save_results = True
pipe3.save_results = True
# run Image3
pipe3.run('starfield_50star4ptdither_no_outlier_asnfile.json')
print('Image 3 pipeline finished.')
2022-10-06 01:06:12,344 - stpipe.Image3Pipeline - INFO - Image3Pipeline instance created.
2022-10-06 01:06:12,346 - stpipe.Image3Pipeline.assign_mtwcs - INFO - AssignMTWcsStep instance created.
2022-10-06 01:06:12,347 - stpipe.Image3Pipeline.tweakreg - INFO - TweakRegStep instance created.
2022-10-06 01:06:12,349 - stpipe.Image3Pipeline.skymatch - INFO - SkyMatchStep instance created.
2022-10-06 01:06:12,350 - stpipe.Image3Pipeline.outlier_detection - INFO - OutlierDetectionStep instance created.
2022-10-06 01:06:12,352 - stpipe.Image3Pipeline.resample - INFO - ResampleStep instance created.
2022-10-06 01:06:12,353 - stpipe.Image3Pipeline.source_catalog - INFO - SourceCatalogStep instance created.
2022-10-06 01:06:12,570 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline running with args ('starfield_50star4ptdither_no_outlier_asnfile.json',).
2022-10-06 01:06:12,576 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'steps': {'assign_mtwcs': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'assign_mtwcs', 'search_output_file': True, 'input_dir': ''}, 'tweakreg': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'sharplo': 0.2, 'sharphi': 1.0, 'roundlo': -1.0, 'roundhi': 1.0, 'brightest': 200, 'peakmax': None, 'bkg_boxsize': 400, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 5, 'searchrad': 2.0, 'use2dhist': False, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'shift', 'nclip': 3, 'sigma': 3, 'abs_refcat': '', 'save_abs_catalog': False, 'abs_minobj': 15, 'abs_searchrad': 6.0, 'abs_use2dhist': True, 'abs_separation': 0.1, 'abs_tolerance': 0.7, 'abs_fitgeometry': 'rshift', 'abs_nclip': 3, 'abs_sigma': 3.0}, 'skymatch': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}, 'outlier_detection': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': True, 'suffix': None, 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None, 'in_memory': False}, 'resample': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}, 'source_catalog': {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 1000, 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'npixels': 25, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}}}
2022-10-06 01:06:12,710 - stpipe.Image3Pipeline - INFO - Prefetching reference files for dataset: 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits' reftypes = ['abvegaoffset', 'apcorr', 'drizpars']
2022-10-06 01:06:12,712 - stpipe.Image3Pipeline - INFO - Prefetch for ABVEGAOFFSET reference file is '/grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf'.
2022-10-06 01:06:12,714 - stpipe.Image3Pipeline - INFO - Prefetch for APCORR reference file is '/grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits'.
2022-10-06 01:06:12,715 - stpipe.Image3Pipeline - INFO - Prefetch for DRIZPARS reference file is '/grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits'.
2022-10-06 01:06:12,716 - stpipe.Image3Pipeline - INFO - Starting calwebb_image3 ...
2022-10-06 01:06:13,381 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg running with args (<ModelContainer>,).
2022-10-06 01:06:13,384 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': True, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'save_catalogs': False, 'catalog_format': 'ecsv', 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'sharplo': 0.2, 'sharphi': 1.0, 'roundlo': -1.0, 'roundhi': 1.0, 'brightest': 200, 'peakmax': None, 'bkg_boxsize': 400, 'enforce_user_order': False, 'expand_refcat': False, 'minobj': 5, 'searchrad': 2.0, 'use2dhist': False, 'separation': 1.0, 'tolerance': 0.7, 'xoffset': 0.0, 'yoffset': 0.0, 'fitgeometry': 'shift', 'nclip': 3, 'sigma': 3, 'abs_refcat': '', 'save_abs_catalog': False, 'abs_minobj': 15, 'abs_searchrad': 6.0, 'abs_use2dhist': True, 'abs_separation': 0.1, 'abs_tolerance': 0.7, 'abs_fitgeometry': 'rshift', 'abs_nclip': 3, 'abs_sigma': 3.0}
2022-10-06 01:06:13,521 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 200 sources in starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:06:13,680 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 200 sources in starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:06:13,832 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 183 sources in starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:06:13,974 - stpipe.Image3Pipeline.tweakreg - INFO - Detected 179 sources in starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits.
2022-10-06 01:06:13,994 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:13,995 - stpipe.Image3Pipeline.tweakreg - INFO - Number of image groups to be aligned: 4.
2022-10-06 01:06:13,995 - stpipe.Image3Pipeline.tweakreg - INFO - Image groups:
2022-10-06 01:06:14,020 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:06:14,021 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:06:14,045 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:06:14,046 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:06:14,071 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:06:14,071 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:06:14,097 - stpipe.Image3Pipeline.tweakreg - INFO - * Images in GROUP 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal':
2022-10-06 01:06:14,098 - stpipe.Image3Pipeline.tweakreg - INFO - starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal
2022-10-06 01:06:14,098 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,098 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,099 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() started on 2022-10-06 01:06:14.098723
2022-10-06 01:06:14,099 - stpipe.Image3Pipeline.tweakreg - INFO - Version 0.8.0
2022-10-06 01:06:14,099 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,227 - stpipe.Image3Pipeline.tweakreg - INFO - Selected image 'GROUP ID: starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' as reference image
2022-10-06 01:06:14,233 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal' to the reference catalog.
2022-10-06 01:06:14,356 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal' catalog with sources from the reference 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' catalog.
2022-10-06 01:06:14,358 - stpipe.Image3Pipeline.tweakreg - INFO - Found 91 matches for 'GROUP ID: starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal'...
2022-10-06 01:06:14,358 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'shift' fit
2022-10-06 01:06:14,360 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'shift' fit for GROUP ID: starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal:
2022-10-06 01:06:14,361 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: 0.0187591 YSH: 0.00410462
2022-10-06 01:06:14,361 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,362 - stpipe.Image3Pipeline.tweakreg - INFO - FIT RMSE: 0.32968 FIT MAE: 0.182423
2022-10-06 01:06:14,362 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 91 objects.
2022-10-06 01:06:14,401 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal' to the reference catalog.
2022-10-06 01:06:14,528 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal' catalog with sources from the reference 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' catalog.
2022-10-06 01:06:14,530 - stpipe.Image3Pipeline.tweakreg - INFO - Found 79 matches for 'GROUP ID: starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal'...
2022-10-06 01:06:14,531 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'shift' fit
2022-10-06 01:06:14,533 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'shift' fit for GROUP ID: starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal:
2022-10-06 01:06:14,533 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: -0.0039417 YSH: 0.00507471
2022-10-06 01:06:14,534 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,534 - stpipe.Image3Pipeline.tweakreg - INFO - FIT RMSE: 0.323486 FIT MAE: 0.171732
2022-10-06 01:06:14,535 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 79 objects.
2022-10-06 01:06:14,575 - stpipe.Image3Pipeline.tweakreg - INFO - Aligning image catalog 'GROUP ID: starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal' to the reference catalog.
2022-10-06 01:06:14,701 - stpipe.Image3Pipeline.tweakreg - INFO - Matching sources from 'starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal' catalog with sources from the reference 'starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal' catalog.
2022-10-06 01:06:14,703 - stpipe.Image3Pipeline.tweakreg - INFO - Found 74 matches for 'GROUP ID: starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal'...
2022-10-06 01:06:14,703 - stpipe.Image3Pipeline.tweakreg - INFO - Performing 'shift' fit
2022-10-06 01:06:14,705 - stpipe.Image3Pipeline.tweakreg - INFO - Computed 'shift' fit for GROUP ID: starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal:
2022-10-06 01:06:14,706 - stpipe.Image3Pipeline.tweakreg - INFO - XSH: -0.0599251 YSH: -0.0277362
2022-10-06 01:06:14,706 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,707 - stpipe.Image3Pipeline.tweakreg - INFO - FIT RMSE: 0.3174 FIT MAE: 0.203801
2022-10-06 01:06:14,707 - stpipe.Image3Pipeline.tweakreg - INFO - Final solution based on 74 objects.
2022-10-06 01:06:14,747 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,747 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() ended on 2022-10-06 01:06:14.747229
2022-10-06 01:06:14,748 - stpipe.Image3Pipeline.tweakreg - INFO - ***** tweakwcs.imalign.align_wcs() TOTAL RUN TIME: 0:00:00.648506
2022-10-06 01:06:14,748 - stpipe.Image3Pipeline.tweakreg - INFO -
2022-10-06 01:06:14,883 - stpipe.Image3Pipeline.tweakreg - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/astropy/modeling/fitting.py:789: AstropyUserWarning: The fit may be poorly conditioned
warnings.warn("The fit may be poorly conditioned\n",
2022-10-06 01:06:14,894 - stpipe.Image3Pipeline.tweakreg - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:06:14,894 - stpipe.Image3Pipeline.tweakreg - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:06:15,004 - stpipe.Image3Pipeline.tweakreg - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:06:15,004 - stpipe.Image3Pipeline.tweakreg - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:06:15,114 - stpipe.Image3Pipeline.tweakreg - WARNING - Failed to update 'meta.wcsinfo' with FITS SIP approximation. Reported error is:
2022-10-06 01:06:15,115 - stpipe.Image3Pipeline.tweakreg - WARNING - "Failed to achieve required error tolerance"
2022-10-06 01:06:15,121 - stpipe.Image3Pipeline.tweakreg - INFO - Step tweakreg done
2022-10-06 01:06:15,348 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch running with args (<ModelContainer>,).
2022-10-06 01:06:15,350 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': '', 'skymethod': 'match', 'match_down': True, 'subtract': False, 'stepsize': None, 'skystat': 'mode', 'dqbits': '~DO_NOT_USE+NON_SCIENCE', 'lower': None, 'upper': None, 'nclip': 5, 'lsigma': 4.0, 'usigma': 4.0, 'binwidth': 0.1}
2022-10-06 01:06:15,467 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:06:15,468 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() started on 2022-10-06 01:06:15.467597
2022-10-06 01:06:15,469 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:06:15,469 - stpipe.Image3Pipeline.skymatch - INFO - Sky computation method: 'match'
2022-10-06 01:06:15,469 - stpipe.Image3Pipeline.skymatch - INFO - Sky matching direction: DOWN
2022-10-06 01:06:15,470 - stpipe.Image3Pipeline.skymatch - INFO - Sky subtraction from image data: OFF
2022-10-06 01:06:15,470 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:06:15,471 - stpipe.Image3Pipeline.skymatch - INFO - ---- Computing differences in sky values in overlapping regions.
2022-10-06 01:06:18,038 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq1_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0.0162815
2022-10-06 01:06:18,039 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq2_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0.0246113
2022-10-06 01:06:18,039 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq3_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0.0157997
2022-10-06 01:06:18,040 - stpipe.Image3Pipeline.skymatch - INFO - * Image ID=starfield_50star4ptdither_seq4_MIRIMAGE_F1130Wexp1_cal.fits. Sky background: 0
2022-10-06 01:06:18,040 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:06:18,040 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() ended on 2022-10-06 01:06:18.040529
2022-10-06 01:06:18,041 - stpipe.Image3Pipeline.skymatch - INFO - ***** jwst.skymatch.skymatch.match() TOTAL RUN TIME: 0:00:02.572932
2022-10-06 01:06:18,041 - stpipe.Image3Pipeline.skymatch - INFO -
2022-10-06 01:06:18,337 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_no_outlier_0_skymatch.fits
2022-10-06 01:06:18,651 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_no_outlier_1_skymatch.fits
2022-10-06 01:06:18,949 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_no_outlier_2_skymatch.fits
2022-10-06 01:06:19,247 - stpipe.Image3Pipeline.skymatch - INFO - Saved model in starfield_50star4ptdither_combined_no_outlier_3_skymatch.fits
2022-10-06 01:06:19,248 - stpipe.Image3Pipeline.skymatch - INFO - Step skymatch done
2022-10-06 01:06:19,633 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection running with args (<ModelContainer>,).
2022-10-06 01:06:19,636 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': True, 'suffix': 'crf', 'search_output_file': False, 'input_dir': '', 'weight_type': 'ivm', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'nlow': 0, 'nhigh': 0, 'maskpt': 0.7, 'grow': 1, 'snr': '5.0 4.0', 'scale': '1.2 0.7', 'backg': 0.0, 'save_intermediate_results': False, 'resample_data': True, 'good_bits': '~DO_NOT_USE', 'scale_detection': False, 'allowed_memory': None, 'in_memory': False}
2022-10-06 01:06:19,637 - stpipe.Image3Pipeline.outlier_detection - INFO - Step skipped.
2022-10-06 01:06:19,646 - stpipe.Image3Pipeline.outlier_detection - INFO - Step outlier_detection done
2022-10-06 01:06:19,999 - stpipe.Image3Pipeline.resample - INFO - Step resample running with args (<ModelContainer>,).
2022-10-06 01:06:20,001 - stpipe.Image3Pipeline.resample - INFO - Step resample parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'i2d', 'search_output_file': True, 'input_dir': '', 'pixfrac': 1.0, 'kernel': 'square', 'fillval': 'INDEF', 'weight_type': 'ivm', 'output_shape': None, 'crpix': None, 'crval': None, 'rotation': None, 'pixel_scale_ratio': 1.0, 'pixel_scale': None, 'single': False, 'blendheaders': True, 'allowed_memory': None, 'in_memory': True}
2022-10-06 01:06:20,020 - stpipe.Image3Pipeline.resample - INFO - Drizpars reference file: /grp/crds/cache/references/jwst/jwst_miri_drizpars_0001.fits
2022-10-06 01:06:20,042 - stpipe.Image3Pipeline.resample - INFO - Driz parameter kernel: square
2022-10-06 01:06:20,043 - stpipe.Image3Pipeline.resample - INFO - Driz parameter pixfrac: 1.0
2022-10-06 01:06:20,043 - stpipe.Image3Pipeline.resample - INFO - Driz parameter fillval: INDEF
2022-10-06 01:06:20,043 - stpipe.Image3Pipeline.resample - INFO - Driz parameter weight_type: ivm
2022-10-06 01:06:20,191 - stpipe.Image3Pipeline.resample - INFO - Blending metadata for starfield_50star4ptdither_combined_no_outlier.fits
2022-10-06 01:06:20,840 - stpipe.Image3Pipeline.resample - INFO - Resampling science data
2022-10-06 01:06:21,471 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:22,487 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:23,523 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:24,641 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:24,829 - stpipe.Image3Pipeline.resample - INFO - Resampling var_rnoise
2022-10-06 01:06:25,434 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:26,470 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:27,429 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:28,399 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:28,650 - stpipe.Image3Pipeline.resample - INFO - Resampling var_poisson
2022-10-06 01:06:29,278 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:30,273 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:31,249 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:32,238 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:32,437 - stpipe.Image3Pipeline.resample - INFO - Resampling var_flat
2022-10-06 01:06:33,078 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:34,067 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:35,119 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:36,091 - stpipe.Image3Pipeline.resample - INFO - Drizzling (1024, 1032) --> (1142, 1118)
2022-10-06 01:06:36,289 - stpipe.Image3Pipeline.resample - WARNING - /internal/data1/jenkins/workspace/Notebooks/jwst_validation_notebooks_spacetelescope/miniconda3/envs/jwst_validation_notebooks/lib/python3.9/site-packages/jwst/resample/resample.py:308: RuntimeWarning: divide by zero encountered in reciprocal
output_variance = np.reciprocal(inverse_variance_sum)
2022-10-06 01:06:36,330 - stpipe.Image3Pipeline.resample - INFO - Update S_REGION to POLYGON ICRS 0.021864896 -0.020260192 0.024938345 0.014762483 359.990651697 0.017771342 359.987578248 -0.017251333
2022-10-06 01:06:36,809 - stpipe.Image3Pipeline.resample - INFO - Saved model in starfield_50star4ptdither_combined_no_outlier_i2d.fits
2022-10-06 01:06:36,810 - stpipe.Image3Pipeline.resample - INFO - Step resample done
2022-10-06 01:06:37,081 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog running with args (<ImageModel(1142, 1118) from starfield_50star4ptdither_combined_no_outlier_i2d.fits>,).
2022-10-06 01:06:37,083 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': 'cat', 'search_output_file': True, 'input_dir': '', 'bkg_boxsize': 1000, 'kernel_fwhm': 3.762, 'snr_threshold': 100, 'npixels': 25, 'deblend': False, 'aperture_ee1': 30, 'aperture_ee2': 50, 'aperture_ee3': 70, 'ci1_star_threshold': 2.0, 'ci2_star_threshold': 1.8}
2022-10-06 01:06:37,101 - stpipe.Image3Pipeline.source_catalog - INFO - Using APCORR reference file: /grp/crds/cache/references/jwst/jwst_miri_apcorr_0005.fits
2022-10-06 01:06:37,111 - stpipe.Image3Pipeline.source_catalog - INFO - Using ABVEGAOFFSET reference file: /grp/crds/cache/references/jwst/jwst_miri_abvegaoffset_0001.asdf
2022-10-06 01:06:37,112 - stpipe.Image3Pipeline.source_catalog - INFO - Instrument: MIRI
2022-10-06 01:06:37,112 - stpipe.Image3Pipeline.source_catalog - INFO - Detector: MIRIMAGE
2022-10-06 01:06:37,113 - stpipe.Image3Pipeline.source_catalog - INFO - Filter: F1130W
2022-10-06 01:06:37,113 - stpipe.Image3Pipeline.source_catalog - INFO - Subarray: FULL
2022-10-06 01:06:37,167 - stpipe.Image3Pipeline.source_catalog - INFO - AB to Vega magnitude offset 5.49349
2022-10-06 01:06:37,246 - stpipe.Image3Pipeline.source_catalog - INFO - Background could not be estimated in meshes. Using the entire unmasked array for background estimation: bkg_boxsize=(1142, 1118).
2022-10-06 01:06:37,822 - stpipe.Image3Pipeline.source_catalog - INFO - Detected 46 sources
2022-10-06 01:06:37,965 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote source catalog: starfield_50star4ptdither_combined_no_outlier_cat.ecsv
2022-10-06 01:06:38,099 - stpipe.Image3Pipeline.source_catalog - INFO - Saved model in starfield_50star4ptdither_combined_no_outlier_segm.fits
2022-10-06 01:06:38,100 - stpipe.Image3Pipeline.source_catalog - INFO - Wrote segmentation map: starfield_50star4ptdither_combined_no_outlier_segm.fits
2022-10-06 01:06:38,102 - stpipe.Image3Pipeline.source_catalog - INFO - Step source_catalog done
2022-10-06 01:06:38,103 - stpipe.Image3Pipeline - INFO - Step Image3Pipeline done
Image 3 pipeline finished.
# read in i2d file without outlier_detection run.
im_i2d2 = ImageModel('starfield_50star4ptdither_combined_no_outlier_i2d.fits')
#norm = ImageNormalize(im_i2d2.data, interval=MinMaxInterval(),
# stretch=LinearStretch())
# mask out DO_NOT_USE values where data has been set to 0 in the combined image
masked_im2 = np.ma.masked_where((im_i2d2.data == 0), im_i2d2.data)
cmap = matplotlib.cm.get_cmap("Greys").copy() # Can be any colormap that you want after the cm
cmap.set_bad(color='white') # color to mark all DO_NOT_USE pixels
plt.figure(figsize=(15,15))
#plt.imshow(im_i2d2.data,origin='lower',norm=norm,vmin=-5, vmax=4)
plt.imshow(masked_im2, origin='lower', cmap = cmap, vmin=5, vmax=20)
plt.colorbar()
<matplotlib.colorbar.Colorbar at 0x7f93b3caa310>
# Compare to image with outlier_detection run
#norm = ImageNormalize(im_i2d.data, interval=MinMaxInterval(),
# stretch=LinearStretch())
plt.figure(figsize=(15,15))
#plt.imshow(im_i2d.data,origin='lower',norm=norm,vmin=-5, vmax=4)
plt.imshow(masked_im, origin='lower', cmap = cmap, vmin=5, vmax=20)
plt.colorbar()
<matplotlib.colorbar.Colorbar at 0x7f93b5875df0>
If the inserted cosmic rays were flagged by the outlier_detection step (no AssertionError) and the outlier_detection run version of the combined image appears cleaner with fewer to no cosmic rays showing, the test passes. If there are only one or two sources that were not properly flagged, they could have been located on a source or been otherwise difficult to detect. Check the locations of the outliers as shown in earlier parts of the notebook to see if any outliers might be likely to fail to be detected.
Author: Misty Cracraft, Senior Staff Scientist, MIRI Branch
Updated On: 02/17/2021